Mcmicro update #28

melonora · 2023-04-02T00:27:02Z

This PR updates the code related to creating a SpatialData object out of MCMICRO output. In particular with this PR, tissue micro array output of MCMICRO is supported in addition to the WSI output. Also some code fixes for exemplar_001 are included

One bug has been fixed when passing on image_models_kwargs. When passing dims, this led to problems as the same kwargs were used for the label model. While the image model for MCMICRO should have dims ('c', 'y', 'x') the label model should have dims ('y', 'x').

For tma the config file is read to check whether the data refers to whole slide imaging or tissue micro array.

Currently, for transformations only global with an Identity transform is possible. This can change later. Furthermore, illumination images and dearray masks are now included in the SpatialData object.

Code has been tested on exemplar-001 and exemplar-002 output of MCMICRO.

- Allow for microarrays - Allow for any segmentation method

codecov-commenter · 2023-04-02T00:28:36Z

Codecov Report

Attention: 122 lines in your changes are missing coverage. Please review.

Comparison is base (755d475) 41.92% compared to head (7e91a66) 37.56%.
Report is 49 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #28      +/-   ##
==========================================
- Coverage   41.92%   37.56%   -4.36%     
==========================================
  Files          16       14       -2     
  Lines         854     1001     +147     
==========================================
+ Hits          358      376      +18     
- Misses        496      625     +129

Files	Coverage Δ
src/spatialdata_io/_constants/_constants.py	`100.00% <100.00%> (ø)`
src/spatialdata_io/readers/mcmicro.py	`16.33% <6.87%> (-28.11%)`	⬇️

... and 6 files with indirect coverage changes

LucaMarconato · 2023-04-02T17:56:43Z

Unfortunately I am not able to test the code at the moment. I can't run the pipeline on exemplar-002 because of a incompatibility with M1 chips labsyspharm/mcmicro#353.

I tried running mc-micro on a linux machine I have remote access to but I got some different errors, and since I only have rootless access, I could not fix them in a straightforward way.

I have added instructions on how to run MCMICRO on the exemplar-001 and exemplar-002 datasets here https://github.com/giovp/spatialdata-sandbox/tree/main/mcmicro_exemplar_io, so someone else can try this. Otherwise I will try again after the paper is out, when I will give more focus on the io scripts.

LucaMarconato · 2023-04-02T17:57:56Z

Anyway, from the code points of view, the changes seems great to me.

melonora · 2023-04-04T09:12:40Z

For trying to get around the problems with the M1 chip, @FloWuenne mentioned that having this in your .bashrc might help:
export DOCKER_DEFAULT_PLATFORM=linux/amd64

LucaMarconato · 2023-07-23T13:32:36Z

Thanks @melonora for the work! This together with the Docker support from Flo it's amazing! I will test it on my machine, hopefully this time I will not have M1 related problems in running MCMICRO 🤞

melonora · 2023-07-23T13:40:59Z

Thanks, can you also particularly check visualization using the view and scatterwidget in napari spatialdata please?

I will pick up the CLI support as well.

FloWuenne · 2023-07-23T18:15:10Z

If you run into problems with M1 @LucaMarconato , try out Gitpod for any python features that don't require visualization (so anything outside of napari). Gitpod runs on Ubuntu, so should not face any M1 issues! I will open a PR for a Github action for the docker container and then adjust the Gitpod PR to make it point toward that container!

LucaMarconato · 2023-07-31T15:29:27Z

@melonora @FloWuenne I just tried installing nextflow in my M1 machine and it seems not be compatible with the latest Java M1 version (I didn't try with the Java Intel version because I prefer to install Apple Silicon native tools). I reached out to Wouter via chat, I think a fast approach would be if you could upload the output data somewhere in GDrive/S3, or pass it to me with an hard drive.

LucaMarconato · 2024-01-06T13:56:26Z

src/spatialdata_io/readers/mcmicro.py

        table[McmicroKeys.INSTANCE_KEY] = "core_" + sample_id + "_" + table[McmicroKeys.INSTANCE_KEY].astype(str)
-    table.index = table[McmicroKeys.INSTANCE_KEY]


I removed this because the index of the table is never used and this could confuse the user.

LucaMarconato · 2024-01-06T13:57:58Z

src/spatialdata_io/readers/mcmicro.py

@@ -93,37 +94,47 @@ def mcmicro(
            marker_names,
        )

+    # in exemplar-001 the raw images are aligned with the illumination images, not with the registration image


the illumination images (in the case of exemplar-001 we have 6 of them, 2 for each cycle), are not aligned with the registration image. They are instead with the raw images. Each there are 3 cycles (=3 raw images), so I create 3 additional coordinate systems.

LucaMarconato · 2024-01-06T15:36:56Z

Thanks Wouter for the great work! I have tested this today on both the exemplar 001 and 002 data and added some changes to fix some transformations and group together the illumination files for a cleaner interface.

Since it has been open for quite a while I am going to merge it now. I have anyway added comments for the fixes that I made directly in the code, please check them out 😊

Finally, here is a GIF showing napari for exemplar-002 (CC @FloWuenne).

LucaMarconato · 2024-01-06T15:39:10Z

src/spatialdata_io/_constants/_constants.py

-    LABELS_PREFIX = "unmicst-"
+    ILLUMINATION_DIR = "illumination"
+    PARAMS_FILE = "qc/params.yml"
+    RAW_DIR = "raw"


I have added this.

LucaMarconato · 2024-01-06T15:39:47Z

src/spatialdata_io/_constants/_constants.py


    # metadata
    COORDS_X = "X_centroid"
    COORDS_Y = "Y_centroid"
    INSTANCE_KEY = "CellID"
+    ILLUMINATION_SUFFIX_DFP = "-dfp"
+    ILLUMINATION_SUFFIX_FFP = "-ffp"


I needed this to recover the name of the coordinate system of the raw images given the filename of illumination files (which ends in -dfp or -ffp.

LucaMarconato · 2024-01-06T15:47:18Z

src/spatialdata_io/readers/mcmicro.py


 from spatialdata_io._constants._constants import McmicroKeys

 __all__ = ["mcmicro"]


+def _get_transformation(


This function is to align the TMAs to the global image.

LucaMarconato · 2024-01-06T15:50:08Z

src/spatialdata_io/readers/mcmicro.py


    return SpatialData(images=images, labels=labels, table=table)


-def _get_images(


I removed this because I needed to know the resolutions of the TMA images in order to compute the transformations to the global view. Also I wanted better control on squeeze and on the dims parameter.

FloWuenne · 2024-01-06T16:57:36Z

Wonderful news this got merged! We will add this as an output option to MCMICRO in nf-core soon I am sure 🤩!

melonora added 4 commits April 1, 2023 17:41

Adjust image parsing to allow tma

dbd0f48

precommit adjustments

f3ab052

Change way of parsing labels

26f5925

- Allow for microarrays - Allow for any segmentation method

Adjust table parsing for TMA MCMICRO output

6b031b2

melonora added 11 commits July 20, 2023 08:30

Merge branch 'scverse:main' into mcmicro_update

1ac0efa

Add parameter loader

fb31cee

load tma from params

af97077

fix regex pattern

f056419

Fix categorical warning

313b435

Force default coordinate_system

61f7a40

add illumnation images

244c747

Add forced transform and resolve future warning

3982a8b

Fix add channel_names

b517c63

Add dearray masks

7ad4cd0

Fix type hint error

315444e

FloWuenne mentioned this pull request Jul 21, 2023

Added first simple version of CLI script. #72

Draft

2 tasks

LucaMarconato mentioned this pull request Oct 10, 2023

mcmicro_io.zip cell and nucleus bitmasks are the same scverse/spatialdata-notebooks#52

Closed

fix mcmicro exemplar-001: fix illumination, add raw

c4a0a4f

LucaMarconato reviewed Jan 6, 2024

View reviewed changes

LucaMarconato approved these changes Jan 6, 2024

View reviewed changes

LucaMarconato reviewed Jan 6, 2024

View reviewed changes

LucaMarconato added 2 commits January 6, 2024 16:40

added tranformations to global image for tmas

c9bc36a

fixed pre-commit

c4c996f

LucaMarconato reviewed Jan 6, 2024

View reviewed changes

LucaMarconato added 2 commits January 6, 2024 16:48

removed unused argument from docstring

4cc00a8

removed commented code

2285a4d

LucaMarconato reviewed Jan 6, 2024

View reviewed changes

LucaMarconato added 2 commits January 6, 2024 16:50

remove more commented code

8c258e4

cleanup

7e91a66

LucaMarconato merged commit fa07ba3 into scverse:main Jan 6, 2024
6 checks passed

LucaMarconato linked an issue Jan 6, 2024 that may be closed by this pull request

mcmicro_io.zip cell and nucleus bitmasks are the same scverse/spatialdata-notebooks#52

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mcmicro update #28

Mcmicro update #28

melonora commented Apr 2, 2023 •

edited

Loading

codecov-commenter commented Apr 2, 2023 •

edited

Loading

LucaMarconato commented Apr 2, 2023 •

edited

Loading

LucaMarconato commented Apr 2, 2023

melonora commented Apr 4, 2023

LucaMarconato commented Jul 23, 2023

melonora commented Jul 23, 2023

FloWuenne commented Jul 23, 2023

LucaMarconato commented Jul 31, 2023

LucaMarconato Jan 6, 2024

LucaMarconato Jan 6, 2024

LucaMarconato commented Jan 6, 2024

LucaMarconato Jan 6, 2024

LucaMarconato Jan 6, 2024

LucaMarconato Jan 6, 2024

LucaMarconato Jan 6, 2024

FloWuenne commented Jan 6, 2024

		table[McmicroKeys.INSTANCE_KEY] = "core_" + sample_id + "_" + table[McmicroKeys.INSTANCE_KEY].astype(str)
		table.index = table[McmicroKeys.INSTANCE_KEY]


		return SpatialData(images=images, labels=labels, table=table)


		def _get_images(

Mcmicro update #28

Mcmicro update #28

Conversation

melonora commented Apr 2, 2023 • edited Loading

codecov-commenter commented Apr 2, 2023 • edited Loading

Codecov Report

LucaMarconato commented Apr 2, 2023 • edited Loading

LucaMarconato commented Apr 2, 2023

melonora commented Apr 4, 2023

LucaMarconato commented Jul 23, 2023

melonora commented Jul 23, 2023

FloWuenne commented Jul 23, 2023

LucaMarconato commented Jul 31, 2023

LucaMarconato Jan 6, 2024

Choose a reason for hiding this comment

LucaMarconato Jan 6, 2024

Choose a reason for hiding this comment

LucaMarconato commented Jan 6, 2024

LucaMarconato Jan 6, 2024

Choose a reason for hiding this comment

LucaMarconato Jan 6, 2024

Choose a reason for hiding this comment

LucaMarconato Jan 6, 2024

Choose a reason for hiding this comment

LucaMarconato Jan 6, 2024

Choose a reason for hiding this comment

FloWuenne commented Jan 6, 2024

melonora commented Apr 2, 2023 •

edited

Loading

codecov-commenter commented Apr 2, 2023 •

edited

Loading

LucaMarconato commented Apr 2, 2023 •

edited

Loading