Fixes to match previous pipeline

* Split mandatory ccfv2 fibers and cell containing regions combination from ccfv3 / ccfv2 annotation combination
* Add normalization and cell overlapping of Nissl and ISH dataset to match Erö's pipeline.
* Fix glia cell density placement, prevent negative densities.
* Add purely inhibitory regions for Erö's method to match model from 2019.
* Fix masks for cerebellar cortex, molecular and cortex L1 and related test.
* Update README.rst in the app/data/ folders.

Co-authored-by: drodarie <[email protected]>

atlas_densities/app/cell_densities.py

@@ -170,7 +170,7 @@ def cell_density(annotation_path, hierarchy_path, nissl_path, output_path):
         _get_voxel_volume_in_mm3(annotation),
         nissl.raw,
     )
-    nissl.with_data(np.asarray(overall_cell_density, dtype=float)).save_nrrd(output_path)
+    nissl.with_data(overall_cell_density).save_nrrd(output_path)
 
 
 @app.command()

atlas_densities/app/combination.py

@@ -52,15 +52,15 @@ def app(verbose):
 )
 @click.option("--output-path", required=True, help="Path of the nrrd file to write")
 @log_args(L)
-def combine_annotations(
+def combine_v2_v3_annotations(
     hierarchy,
     brain_annotation_ccfv2,
     fiber_annotation_ccfv2,
     brain_annotation_ccfv3,
     output_path,
 ):
     # pylint: disable=line-too-long
-    """Generate and save the combined annotation file.
+    """Generate and save the v2-v3 combined annotation file.
 
     The annotation file `brain_annotation_ccfv3` is the annotation file containing
     the least complete annotation. There are two use cases: a resolution of 10 or 25 um.
@@ -102,6 +102,56 @@ def combine_annotations(
     combined_annotation.save_nrrd(output_path)
 
 
+@app.command()
+@click.option(
+    "--brain-annotation-ccfv2",
+    type=EXISTING_FILE_PATH,
+    required=True,
+    help=("This brain annotation file contains the most complete annotation."),
+)
+@click.option(
+    "--fiber-annotation-ccfv2",
+    type=EXISTING_FILE_PATH,
+    required=True,
+    help="Fiber annotation is not included in the CCF-v2 2011 annotation files.",
+)
+@click.option("--output-path", required=True, help="Path of the nrrd file to write")
+@log_args(L)
+def combine_ccfv2_annotations(
+    brain_annotation_ccfv2,
+    fiber_annotation_ccfv2,
+    output_path,
+):
+    # pylint: disable=line-too-long
+    """Generate and save the ccfv2 combined annotation file.
+
+    The ccfv2 annotations are split into two volumes. `fiber_annotation_ccfv2` describes solely the
+    fibers and ventricular related regions while `brain_annotation_ccfv2` contains all other brain
+    regions. There are two use cases: a resolution of 10 or 25 um.
+
+    For a resolution of 10 um, the path arguments should be the following:
+
+    \b
+    - `brain_annotation_ccfv2` is the path to a copy of
+    ``AIBS_ANNOTATION_URL``/mouse_2011/annotation_10.nrrd
+
+    \b
+    - `fiber_annotation` is the path to a copy of
+    ``AIBS_ANNOTATION_URL``/mouse_2011/annotationFiber_10_2011.nrrd
+
+    where ``AIBS_ANNOTATION_URL`` is
+    http://download.alleninstitute.org/informatics-archive/current-release/mouse_ccf/annotation.
+    """
+
+    brain_annotation_ccfv2 = voxcell.VoxelData.load_nrrd(brain_annotation_ccfv2)
+    fiber_annotation_ccfv2 = voxcell.VoxelData.load_nrrd(fiber_annotation_ccfv2)
+
+    combined_annotation = annotations_combinator.combine_ccfv2_annotations(
+        brain_annotation_ccfv2, fiber_annotation_ccfv2
+    )
+    combined_annotation.save_nrrd(output_path)
+
+
 @app.command()
 @common_atlas_options
 @click.option(

atlas_densities/app/data/markers/README.rst

@@ -4,38 +4,56 @@ Description
 
 This folder contains data pertaining to gene marker volumes provided by AIBS.
 
-The file `realigned_slices.json` contains the slice indices of several gene marker volumetric files which have been
-realigned manually by Csaba Eroe (former BBP PHD student). It has the form of a dictionary whose keys are AIBS dataset identifiers
-(see the AIBS find queries of `Gene Search`_) and whose values are lists of integers.
+The file `realigned_slices_XXX.json` contains position indices of the original image slices for
+several AIBS ISH experiments. It has the form of a dictionary whose keys are AIBS dataset
+identifiers (see the AIBS find queries of `Gene Search`_) and whose values are the position indices
+of the slice images with respect to the reference volumes. These position values can be extracted
+from the ISH experiments metadata provided by the AIBS.
 
-The computation of average marker intensities in AIBS brain regions is restricted to these slices which can be trusted.
-Such a computation is the basis of a linear fitting
-on a 2D point cloud (average marker intensity, average cell density) used to estimate cell densities in regions where no
-measurement is available.
+The content of `realigned_slices_XXX.json` depends on which annotation and Nissl nrrd files are
+considered (`_XXX` in the filename). Two versions of this file are provided in this folder, one for
+the AIBS CCFv2 version and one for the CCFbbp version (see `Rodarie et al. (2021)`_ for more
+details). The ISH images from the AIBS are registered to the CCFv2 and CCFv3 brain volume so no
+transformation is required for these reference volume versions. For the CCFbbp, however, some ISH
+slices position are slightly shifted with respect to the Nisslbbp. Hence, a manual selection of the
+matching Nissl slice for each ISH slice is required to obtain the best cell density estimates.
 
-The file `realigned_slices.json` is referred to by the gene marker configuration file,
-an input of the command line `atlas-densities cell-densities fit-average-densities`,
-see `atlas_densities.app.cell_densities`. The gene marker configuration binds
-gene marker volumetric files (nrrd files) with the appropriate slices of `realigned_slices.json`.
-
-The content of `realigned_slices.json` depends on which annotation and Nissl nrrd files are considered.
-For atlas files older than AIBS CCFv2, one should use the file stored in this directory when computing average
-densities. This file is the companion file of a manual re-alignment process conducted by Csaba Eroe (former BBP PHD student).
+The computation of average marker intensities in AIBS brain regions is restricted to the original
+ISH slices, which can be trusted (see `Rodarie et al. (2021)`_). Such a computation is the basis of
+a linear fitting on a 2D point cloud (average marker intensity, average cell density) used to
+estimate inhibitory neuron densities in regions where no literature measurement is available.
 
-For vanilla AIBS CCFV2 or CCFV3 files, `realigned_slices.json` is filled with the indices inferred the AIBS 2D images metadata
-(no BBP processing in this case).
+The file `realigned_slices_XXX.json` is referred to by the gene marker configuration file,
+an input of the command line `atlas-densities cell-densities fit-average-densities`,
+see `atlas_densities.app.cell_densities`. The gene marker configuration binds gene marker volumetric
+files (nrrd files) with the appropriate slices of `realigned_slices_XXX.json`.
 
-The file `std_cells.json` contains the standard deviation of every AIBS region volumetric cell
-density. This file has been produced by Csabe Eroe, assuming that Nissl stains intensity depends linearily on volumetric cell density across the mouse brain. This file is used
-to assign a "standard deviation" value to every region where a pre-computed volumetric neuron density
-has been used to assign an average neuron density. This holds for every region whose neurons are inhibitory
-only. (Those regions are listed in `atlas_densities/app/data/measurements/homogenous_regions.csv`.)
+The file `std_cells.json` (in `atlas_densities/app/data/measurements/`) contains the cell density
+standard deviation the BBP cell atlas for every AIBS region. This file has been produced by
+Csabe Eroe (see `Eroe et al. (2018)`_), based on multiple Nissl volumes, assuming that Nissl stains
+intensity depends linearly on volumetric cell density across the mouse brain. This file is used to
+assign a "standard deviation" value to every region where a pre-computed volumetric neuron density
+has been used to assign an average neuron density. This holds for every region whose neurons are
+inhibitory only. (Those regions are listed in
+`atlas_densities/app/data/measurements/homogenous_regions.csv`.)
 
 
 Gene markers configuration files
 --------------------------------
-The option `--gene-config-path` of the CLI `atlas-densities cell-densities fit-average-densities` expects
-a path to a yaml file of the following form:
+Combine ISH datasets for glia cells
+>>>>>>>>>>
+
+The option `--config` of the CLI `atlas-densities combination combine-markers` expects a path to a
+yaml file describing the location of the ISH datasets files as well weight parameters to combine
+different genes expressed by glia cell types. `combine_markers_ccfv2_config.yaml` corresponds to
+this configuration file, used to obtain the results of `Eroe et al. (2018)`_, using the CCFv2
+reference volumes.
+
+Fitting of transfer functions from mean region intensity to neuron density
+>>>>>>>>>>
+
+The option `--gene-config-path` of the CLI `atlas-densities cell-densities fit-average-densities`
+expects a path to a yaml file of the following form:
 
 .. code:: yaml
 
@@ -49,10 +67,13 @@ a path to a yaml file of the following form:
         sst: 1001
         vip: 77371835
         gad67: 479
-    realignedSlicesPath: "realigned_slices.json"
+    realignedSlicesPath: "realigned_slices_XXX.json"
     cellDensityStandardDeviationsPath: "std_cells.json"
 
-The sectionDataSetID values are AIBS dataset identifiers recorded in `realigned_slices.json`.
-
+The sectionDataSetID values are AIBS dataset identifiers recorded in `realigned_slices_XXX.json`.
+An example of this configuration file (`fit_average_densities_ccfv2_config.yaml`) is provided for the
+CCFv2 reference volumes.
 
-.. _`Gene Search`: https://mouse.brain-map.org/
+.. _`Gene Search`: https://mouse.brain-map.org/
+.. _`Rodarie et al. (2021)`: https://www.biorxiv.org/content/10.1101/2021.11.20.469384v2
+.. _`Eroe et al. (2018)`: https://www.frontiersin.org/articles/10.3389/fninf.2018.00084/full

atlas_densities/app/data/markers/combine_markers_ccfv2_config.yaml

@@ -0,0 +1,61 @@
+cellType:
+- oligodendrocyte
+- astrocyte
+- microglia
+brainRegion:
+- cerebellum
+- striatum
+gene:
+- cnp
+- mbp
+- gfap
+- s100b
+- aldh1l1
+- tmem119
+inputGeneVolumePath:
+  cnp: data/ccfv2/marker_volumes/CNP.nrrd
+  mbp: data/ccfv2/marker_volumes/MBP.nrrd
+  gfap: data/ccfv2/marker_volumes/GFAP.nrrd
+  s100b: data/ccfv2/marker_volumes/S100b.nrrd
+  aldh1l1: data/ccfv2/marker_volumes/ALDH1L1.nrrd
+  tmem119: data/ccfv2/marker_volumes/TMEM119.nrrd
+outputCellTypeVolumePath:
+  oligodendrocyte: data/ccfv2/relative_marker_volumes/oligodendrocyte.nrrd
+  astrocyte: data/ccfv2/relative_marker_volumes/astrocyte.nrrd
+  microglia: data/ccfv2/relative_marker_volumes/microglia.nrrd
+# Overall glia intensity volume
+outputOverallGliaVolumePath: data/ccfv2/relative_marker_volumes/glia.nrrd
+# The so-called global celltype scaling factors S_celltype of
+# "A Cell Atlas for the Mouse Brain", C. Eroe et al, 2018.
+# These factors are simply the proportions of the different glia cell types
+# in the mouse brain.
+outputCellTypeProportionsPath: data/ccfv2/relative_marker_volumes/glia_proportions.json
+# Cell densities in number of cells per mm^3
+cellDensity:
+  cerebellum:
+    oligodendrocyte: 13750
+    astrocyte: 1512
+    microglia: 8624
+  striatum:
+    oligodendrocyte: 9950
+    astrocyte: 9867
+    microglia: 12100
+combination:
+- cellType: oligodendrocyte
+  gene: cnp
+  averageExpressionIntensity: 35.962800
+- cellType: oligodendrocyte
+  gene: mbp
+  averageExpressionIntensity: 3.304965
+- cellType: astrocyte
+  gene: gfap
+  averageExpressionIntensity: 3.2097903
+- cellType: astrocyte
+  gene: s100b
+  averageExpressionIntensity: 4.174825
+- cellType: astrocyte
+  gene: aldh1l1
+  averageExpressionIntensity: 1.326080
+- cellType: microglia
+  gene: tmem119
+  averageExpressionIntensity: 0.654761

atlas_densities/app/data/markers/fit_average_densities_ccfv2_config.yaml

@@ -0,0 +1,12 @@
+inputGeneVolumePath:
+    pv: data/ccfv2/marker_volumes/pvalb.nrrd
+    sst: data/ccfv2/marker_volumes/SST.nrrd
+    vip: data/ccfv2/marker_volumes/VIP.nrrd
+    gad67: data/ccfv2/marker_volumes/gad1.nrrd
+sectionDataSetID:
+    pv: "868"
+    sst: "1001"
+    vip: "77371835"
+    gad67: "479"
+realignedSlicesPath: atlas_densities/app/measurements/markers/realigned_slices_ccfv2.json
+cellDensityStandardDeviationsPath: atlas_densities/app/measurements/std_cells.json