Coexistance of cellfinder+brainreg CLI tool and cellfinder-core workflow

MANIFEST.in

@@ -9,8 +9,8 @@ exclude *.ini
 
 recursive-include brainglobe_workflows *.py
 recursive-include brainglobe_workflows/configs *.json
-recursive-include brainglobe_benchmarks *.py
-recursive-exclude brainglobe_benchmarks/results *
+recursive-include benchmarks *.py
+recursive-exclude benchmarks/results *
 include asv.conf.json
 
 recursive-exclude * __pycache__

README.md

@@ -16,27 +16,23 @@
 # BrainGlobe Workflows
 
 `brainglobe-workflows` is a package that provides users with a number of out-of-the-box data analysis workflows employed in neuroscience, implemented using BrainGlobe tools.
+
+These workflows represent the most common use-cases and are meant to be easy to reuse. They also serve as an example of how to combine several BrainGlobe tools  (possibly together with other tools) to achieve a goal, such as whole brain cell detection and atlas registration.
+
 You can view the [full documentation for each workflow](https://brainglobe.info/documentation/brainglobe-workflows/index.html) online.
 You can also find the documentation for the backend BrainGlobe tools these workflows use [on our website](https://brainglobe.info/).
 
-At present, the package offers the following workflows:
-
-- [cellfinder](#cellfinder): Whole-brain detection, registration, and analysis.
+At present, the package offers the following workflows to users:
 
-These workflows should be representative of the most common use-cases and are meant to be easy to reuse. They also serve as an example of how to combine several BrainGlobe tools to achieve a goal, such as whole brain cell detection and atlas registration.
-These workflows typically combine several BrainGlobe tools (possibly together with other tools) to achieve a goal,
-such as whole brain cell detection and atlas registration.
+- [cellfinder](#cellfinder): a command-line tool for whole-brain detection, registration, and analysis.
 
-## Secondary purposes of brainglobe-workflows, for developers
+Additionally, this repository provides functionalities to support code developers. See [Developers documentation](#developers-documentation) for further details.
 
-We also use these workflows to support code development. We do this by regularly benchmarking the time they take to complete to ensure performance is stable as the code changes.
-* Developers can install these benchmarks locally via `pip install .[dev]`. By executing `asv run`,  the benchmarks will run with default parameters on a small dataset that is downloaded from [GIN](https://gin.g-node.org/G-Node/info/wiki). See [the asv docs](https://asv.readthedocs.io/en/v0.6.1/using.html#running-benchmarks) for further details on how to run benchmarks.
-* Developers can also run these benchmarks on  data available locally, by specifying the relevant paths in an input configuration file (JSON file).
-* We additionally run and benchmark the workflows locally on a internal desktop machine with large example datasets. These benchmarks are run periodically and the results are made publicly available.
+## User documentation
 
-## Installation
 
-If you want to install BrainGlobe workflows as a standalone tool, you can run `pip install` in your desired environment:
+### Installation of the cellfinder CLI tool
+At the moment, users can install the cellfinder CLI tool as a standalone tool, by running `pip install` in your desired environment:
 
 ```bash
 pip install brainglobe-workflows
@@ -45,32 +41,16 @@ pip install brainglobe-workflows
 `brainglobe-workflows` is built using BrainGlobe tools, and it will automatically fetch the tools that it needs and install them into your environment.
 Once BrainGlobe version 1 is available, this package will fetch all BrainGlobe tools and handle their install into your environment, to prevent potential conflicts from partial-installs.
 
-## Contributing
-
-Contributions to BrainGlobe are more than welcome.
-Please see the [developers guide](https://brainglobe.info/developers/index.html).
-
-## Citing `brainglobe-workflows`
-
-**If you use any tools in the [brainglobe suite](https://brainglobe.info/documentation/index.html), please [let us know](mailto:[email protected]?subject=cellfinder), and we'd be happy to promote your paper/talk etc.**
-
-If you find [`cellfinder`](#cellfinder) useful, and use it in your research, please cite the paper outlining the cell detection algorithm:
-> Tyson, A. L., Rousseau, C. V., Niedworok, C. J., Keshavarzi, S., Tsitoura, C., Cossell, L., Strom, M. and Margrie, T. W. (2021) “A deep learning algorithm for 3D cell detection in whole mouse brain image datasets’ PLOS Computational Biology, 17(5), e1009074
-[https://doi.org/10.1371/journal.pcbi.1009074](https://doi.org/10.1371/journal.pcbi.1009074)
->
-If you use any of the image registration functions in `cellfinder`, please also cite [`brainreg`](https://github.com/brainglobe/brainreg#citing-brainreg).
-
----
 
-## Cellfinder
+### Cellfinder Command Line Interface (CLI)
 
 Whole-brain cell detection, registration and analysis.
 
 If you want to just use the cell detection part of `cellfinder`, please see the standalone [cellfinder-core](https://github.com/brainglobe/cellfinder-core) package, or the [cellfinder plugin](https://github.com/brainglobe/cellfinder-napari) for [napari](https://napari.org/).
 
 `cellfinder` is a collection of tools developed by [Adam Tyson](https://github.com/adamltyson), [Charly Rousseau](https://github.com/crousseau) and [Christian Niedworok](https://github.com/cniedwor) in the [Margrie Lab](https://www.sainsburywellcome.org/web/groups/margrie-lab), generously supported by the [Sainsbury Wellcome Centre](https://www.sainsburywellcome.org/web/).
 
-`cellfinder` is a designed for the analysis of whole-brain imaging data such as [serial-section imaging](https://sainsburywellcomecentre.github.io/OpenSerialSection/) and lightsheet imaging in cleared tissue.
+`cellfinder` is designed for the analysis of whole-brain imaging data such as [serial-section imaging](https://sainsburywellcomecentre.github.io/OpenSerialSection/) and lightsheet imaging in cleared tissue.
 The aim is to provide a single solution for:
 
 - Cell detection (initial cell candidate detection and refinement using  deep learning) (using [cellfinder-core](https://github.com/brainglobe/cellfinder-core)),
@@ -84,3 +64,28 @@ cellfinder -s signal_images -b background_images -o output_dir --metadata metada
 ```
 
 Full documentation can be found [here](https://brainglobe.info/documentation/cellfinder/index.html).
+
+
+## Developer documentation
+
+This repository also includes workflow scripts that are benchmarked to support code development.
+These benchmarks are run regularly to ensure performance is stable, as the tools are developed and extended.
+* Developers can install these benchmarks locally via `pip install .[dev]`. By executing `asv run`, the benchmarks will run with default parameters on a small dataset that is downloaded from [GIN](https://gin.g-node.org/G-Node/info/wiki). See [the asv docs](https://asv.readthedocs.io/en/v0.6.1/using.html#running-benchmarks) for further details on how to run benchmarks.
+* Developers can also run these benchmarks on data they have stored locally, by specifying the relevant paths in an input (JSON) file.
+* We also maintain an internal runner that benchmarks the workflows over a large, exemplar dataset, of the scale we expect users to be handling. The result of these benchmarks are made publicly available.
+
+Contributions to BrainGlobe are more than welcome.
+Please see the [developer guide](https://brainglobe.info/developers/index.html).
+
+
+## Citing `brainglobe-workflows`
+
+**If you use any tools in the [brainglobe suite](https://brainglobe.info/documentation/index.html), please [let us know](mailto:[email protected]?subject=cellfinder), and we'd be happy to promote your paper/talk etc.**
+
+If you find [`cellfinder`](#cellfinder) useful, and use it in your research, please cite the paper outlining the cell detection algorithm:
+> Tyson, A. L., Rousseau, C. V., Niedworok, C. J., Keshavarzi, S., Tsitoura, C., Cossell, L., Strom, M. and Margrie, T. W. (2021) “A deep learning algorithm for 3D cell detection in whole mouse brain image datasets’ PLOS Computational Biology, 17(5), e1009074
+[https://doi.org/10.1371/journal.pcbi.1009074](https://doi.org/10.1371/journal.pcbi.1009074)
+>
+If you use any of the image registration functions in `cellfinder`, please also cite [`brainreg`](https://github.com/brainglobe/brainreg#citing-brainreg).
+
+---

asv.conf.json

@@ -35,7 +35,7 @@
 
     // Customizable commands for installing and uninstalling the project.
     // See asv.conf.json documentation.
-    "install_command": ["in-dir={env_dir} python -mpip install --force-reinstall {wheel_file}"],
+    "install_command": ["in-dir={env_dir} python -mpip install --force-reinstall '{wheel_file}[asv_version]'"],
     "uninstall_command": ["return-code=any python -mpip uninstall -y {project}"],
 
     // List of branches to benchmark. If not provided, defaults to "master"

brainglobe_benchmarks/cellfinder.py → benchmarks/cellfinder_core.py

@@ -7,11 +7,13 @@
 from cellfinder_core.main import main as cellfinder_run
 from cellfinder_core.tools.IO import read_with_dask
 
-from brainglobe_workflows.cellfinder import (
+from brainglobe_workflows.cellfinder_core.cellfinder import (
     CellfinderConfig,
     run_workflow_from_cellfinder_run,
 )
-from brainglobe_workflows.cellfinder import setup as setup_cellfinder_workflow
+from brainglobe_workflows.cellfinder_core.cellfinder import (
+    setup as setup_cellfinder_workflow,
+)
 from brainglobe_workflows.utils import DEFAULT_JSON_CONFIG_PATH_CELLFINDER
 
 

brainglobe_workflows/analyse/analyse.py → brainglobe_workflows/cellfinder_brainreg/analyse/analyse.py

@@ -18,7 +18,9 @@
 from brainglobe_utils.general.system import ensure_directory_exists
 from brainglobe_utils.pandas.misc import safe_pandas_concat, sanitise_df
 
-from brainglobe_workflows.export.export import export_points
+from brainglobe_workflows.cellfinder_brainreg.export.export import (
+    export_points,
+)
 
 
 class Point:

brainglobe_workflows/export/export.py → brainglobe_workflows/cellfinder_brainreg/export/export.py

@@ -1,5 +1,7 @@
-from brainglobe_workflows.export.abc4d import export_points as abc4d_export
-from brainglobe_workflows.export.brainrender import (
+from brainglobe_workflows.cellfinder_brainreg.export.abc4d import (
+    export_points as abc4d_export,
+)
+from brainglobe_workflows.cellfinder_brainreg.export.brainrender import (
     export_points as brainrender_export,
 )
 

brainglobe_workflows/extract/extract_cubes.py → brainglobe_workflows/cellfinder_brainreg/extract/extract_cubes.py

@@ -22,8 +22,10 @@
 from tifffile import tifffile
 from tqdm import tqdm
 
-from brainglobe_workflows.tools import image_processing as img_tools
-from brainglobe_workflows.tools import system
+from brainglobe_workflows.cellfinder_brainreg.tools import (
+    image_processing as img_tools,
+)
+from brainglobe_workflows.cellfinder_brainreg.tools import system
 
 
 class StackSizeError(Exception):

brainglobe_workflows/figures/figures.py → brainglobe_workflows/cellfinder_brainreg/figures/figures.py

@@ -1,6 +1,6 @@
 import logging
 
-from brainglobe_workflows.figures import heatmap
+from brainglobe_workflows.cellfinder_brainreg.figures import heatmap
 
 
 def run(args, atlas, downsampled_shape):

brainglobe_workflows/main.py → brainglobe_workflows/cellfinder_brainreg/main.py

@@ -44,7 +44,7 @@ def main():
     suppress_tf_logging(tf_suppress_log_messages)
     from brainreg.core.main import main as register
 
-    from brainglobe_workflows.tools import prep
+    from brainglobe_workflows.cellfinder_brainreg.tools import prep
 
     start_time = datetime.now()
     args, arg_groups, what_to_run, atlas = prep.prep_cellfinder_general()
@@ -105,9 +105,9 @@ def run_all(args, what_to_run, atlas):
     from cellfinder_core.tools import prep
     from cellfinder_core.tools.IO import read_with_dask
 
-    from brainglobe_workflows.analyse import analyse
-    from brainglobe_workflows.figures import figures
-    from brainglobe_workflows.tools.prep import (
+    from brainglobe_workflows.cellfinder_brainreg.analyse import analyse
+    from brainglobe_workflows.cellfinder_brainreg.figures import figures
+    from brainglobe_workflows.cellfinder_brainreg.tools.prep import (
         prep_candidate_detection,
         prep_channel_specific_general,
     )

brainglobe_workflows/tools/prep.py → brainglobe_workflows/cellfinder_brainreg/tools/prep.py

@@ -18,9 +18,11 @@
 from fancylog import fancylog
 
 import brainglobe_workflows as program_for_log
-import brainglobe_workflows.tools.parser as parser
-from brainglobe_workflows.tools import system, tools
-from brainglobe_workflows.tools.parser import cellfinder_parser
+import brainglobe_workflows.cellfinder_brainreg.tools.parser as parser
+from brainglobe_workflows.cellfinder_brainreg.tools import system, tools
+from brainglobe_workflows.cellfinder_brainreg.tools.parser import (
+    cellfinder_parser,
+)
 
 
 def get_arg_groups(args, parser):

pyproject.toml

@@ -8,7 +8,7 @@ authors = [
 ]
 description = "A collection of end-to-end data analysis workflows executed using BrainGlobe tools."
 readme = "README.md"
-license = { file = "LICENSE" } #{text = "BSD-3-Clause"}
+license = { file = "LICENSE" }
 requires-python = ">=3.9"
 dynamic = ["version"]
 
@@ -24,6 +24,8 @@ classifiers = [
     "Topic :: Scientific/Engineering :: Image Recognition",
 ]
 
+# Below the dependenciess for the cellfinder CLI tool only
+# (i.e., only what users will need for the CLI)
 dependencies = [
     "brainreg>=1.0.0",
     "cellfinder-core>=0.2.4,<1.0.0",
@@ -40,11 +42,11 @@ dependencies = [
     "scikit-image",
     "tifffile",
     "tqdm",
-    "asv",
-    "pooch",
 ]
 
 [project.optional-dependencies]
+# Below, everything developers need to contribute
+# to the workflows (i.e., run tests, run benchmarks, etc)
 dev = [
     "black",
     "pytest-cov",
@@ -53,7 +55,19 @@ dev = [
     "coverage>=5.0.3",
     "pre-commit",
     "setuptools_scm",
+    "asv",
+    "pooch",
 ]
+# Below, all the dependencies asv needs to run the benchmarks
+# (i.e., everything needed to install this package without the CLI tool)
+# Once the cellfinder CLI tool is deprecated, these will move to the
+# default dependencies.
+asv_version = [
+    "asv",
+    "pooch",
+    "cellfinder-core",
+]
+
 napari = [
     "napari[pyqt5]",
     "brainglobe-napari-io",
@@ -67,8 +81,8 @@ napari = [
 "Source Code" = "https://github.com/brainglobe/brainglobe-workflows"
 
 [project.scripts]
-cellfinder-workflow = "brainglobe_workflows.cellfinder:main_app_wrapper"
-cellfinder = "brainglobe_workflows.main:main"
+cellfinder-workflow = "brainglobe_workflows.cellfinder_core.cellfinder:main_app_wrapper"
+cellfinder = "brainglobe_workflows.cellfinder_brainreg.main:main"
 
 [build-system]
 requires = ["setuptools>=45", "wheel", "setuptools_scm[toml]>=6.2"]
@@ -80,7 +94,7 @@ zip-safe = false
 
 [tool.setuptools.packages.find]
 include = ["brainglobe_workflows"]
-exclude = ["tests", "resources"]
+exclude = ["brainglobe_workflows.cellfinder_core","tests", "resources", "benchmarks"] # it's not excluding "brainglobe_workflows.cellfinder_core"!
 
 [tool.black]
 target-version = ["py39", "py310"]
@@ -97,8 +111,11 @@ archs = ["x86_64", "arm64"]
 [tool.check-manifest]
 
 [tool.pytest.ini_options]
-addopts = "--cov=brainglobe_workflows"
-
+addopts = "--cov=brainglobe-workflows"
+markers = [
+    "slow: marks tests as slow (deselect with '-m \"not slow\"')",
+    "serial",
+]
 [tool.ruff]
 line-length = 79
 exclude = ["__init__.py", "build", ".eggs"]
@@ -110,18 +127,28 @@ fix = true
 [tool.tox]
 legacy_tox_ini = """
 [tox]
-envlist = py{38,39,310}
+envlist = py{39,310}-{coredev}
 isolated_build = True
 
 [gh-actions]
 python =
-    3.8: py38
     3.9: py39
     3.10: py310
 
+[gh-actions:env]
+# This runs the coredev environment if the "coredev" github actions input
+# is set to "true"
+INPUT_COREDEV =
+    true: coredev
+
 [testenv]
 extras =
     dev
+deps =
+    coredev: git+https://github.com/brainglobe/cellfinder-core.git
 commands =
-    pytest -v --color=yes --cov=brainglobe_workflows --cov-report=xml
+    pytest {toxinidir} -v --color=yes --cov=./ --cov-report=xml
+description =
+    Run tests
+    coredev: Run tests with the development version of cellfinder-core
 """

tests/test_integration/brainglobe_benchmarks/test_cellfinder.py → tests/benchmarks/test_cellfinder.py

@@ -53,7 +53,7 @@ def asv_config_monkeypatched_path(tmp_path: Path) -> str:
     return str(asv_monkeypatched_path)
 
 
-@pytest.mark.skip(reason="will be worked on a separate PR")
+@pytest.mark.skip(reason="focus of PR32")
 def test_run_benchmarks(asv_config_monkeypatched_path):
     # --- ideally monkeypatch an asv config so that results are in tmp_dir?
 

tests/cellfinder/test_integration/test_detection.py → tests/cellfinder_brainreg/test_integration/test_detection.py

@@ -1,14 +1,17 @@
 import os
 import sys
 from math import isclose
+from pathlib import Path
 
 import brainglobe_utils.IO.cells as cell_io
 import pytest
 
-from brainglobe_workflows.main import main as cellfinder_run
+from brainglobe_workflows.cellfinder_brainreg.main import (
+    main as cellfinder_run,
+)
 
-data_dir = os.path.join(
-    os.getcwd(), "tests", "data", "integration", "detection"
+data_dir = Path(__file__).parents[3] / Path(
+    "tests", "data", "integration", "detection"
 )
 signal_data = os.path.join(data_dir, "crop_planes", "ch0")
 background_data = os.path.join(data_dir, "crop_planes", "ch1")

tests/cellfinder/test_integration/test_extract.py → tests/cellfinder_brainreg/test_integration/test_extract.py

@@ -1,4 +1,5 @@
 import os
+from pathlib import Path
 
 import imio
 import numpy as np
@@ -11,10 +12,11 @@
 from brainglobe_utils.IO.cells import get_cells
 from tifffile import tifffile
 
-import brainglobe_workflows.extract.extract_cubes as extract_cubes
-
-data_dir = os.path.join("tests", "data")
+from brainglobe_workflows.cellfinder_brainreg.extract import (
+    extract_cubes as extract_cubes,
+)
 
+data_dir = Path(__file__).parents[3] / Path("tests", "data")
 signal_data_dir = os.path.join(data_dir, "signal")
 background_data_dir = os.path.join(data_dir, "background")
 xml_path = os.path.join(data_dir, "cube_extract", "cells.xml")