Strategy to ensure alignment of images and derivatives #265

Nilser3 · 2023-09-21T10:24:10Z

Description

WORKING BRANCH: nlm/alignment_images_derivatives

It has been observed that in some subjects of the basel-mp2rage dataset there is a mismatch between the q_form / s_form of the images and the derivatives, which could mean some problems in the visualization of the images.

To solve this problem maybe we should make sure that the q_form and the s_form of image and mask are the same

sct_image -i sub-P001_UNIT1.nii.gz -header

--
Spinal Cord Toolbox (6.0)

sct_image -i sub-P001_UNIT1.nii.gz -header
--

sizeof_hdr      348
data_type       UINT16
dim             [3, 240, 256, 176, 1, 1, 1, 1]
vox_units       mm
time_units      s
datatype        512
nbyper          2
bitpix          16
pixdim          [1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0]
vox_offset      352
cal_max         0.000000
cal_min         0.000000
scl_slope       1.000000
scl_inter       0.000000
phase_dim       0
freq_dim        0
slice_dim       0
slice_name      Unknown
slice_code      0
slice_start     0
slice_end       0
slice_duration  0.000000
toffset         0.000000
intent          Unknown
intent_code     0
intent_name
intent_p1       0.000000
intent_p2       0.000000
intent_p3       0.000000
qform_name      Scanner Anat
qform_code      1
qto_xyz:1       0.000000 0.000000 1.000000 -89.557289
qto_xyz:2       -1.000000 0.000000 0.000000 150.847916
qto_xyz:3       0.000000 -1.000000 0.000000 120.816551
qto_xyz:4       0.000000 0.000000 0.000000 1.000000
qform_xorient   Anterior-to-Posterior
qform_yorient   Superior-to-Inferior
qform_zorient   Left-to-Right
sform_name      Scanner Anat
sform_code      1
sto_xyz:1       -0.000000 -0.000000 1.000000 -89.557289
sto_xyz:2       -1.000000 -0.000000 -0.000000 150.847916
sto_xyz:3       0.000000 -1.000000 0.000000 120.816551
sto_xyz:4       0.000000 0.000000 0.000000 1.000000
sform_xorient   Anterior-to-Posterior
sform_yorient   Superior-to-Inferior
sform_zorient   Left-to-Right
file_type       NIFTI-1+
file_code       1
descrip         MRtrix version: 3.0_RC3-91-gd2cd716d
aux_file

sct_image -i sub-P001_UNIT1_lesion-manualNeuroPoly.nii.gz -header

--
Spinal Cord Toolbox (6.0)

sct_image -i sub-P001_UNIT1_lesion-manualNeuroPoly.nii.gz -header
--

Image header specifies datatype 'uint16', but array is of type 'float64'. Header metadata will be overwritten to use 'float64'.
sizeof_hdr      348
data_type       FLOAT64
dim             [3, 240, 256, 176, 1, 1, 1, 1]
vox_units       mm
time_units      Unknown
datatype        64
nbyper          8
bitpix          64
pixdim          [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
vox_offset      352
cal_max         0.000000
cal_min         0.000000
scl_slope       0.000015
scl_inter       0.000000
phase_dim       0
freq_dim        0
slice_dim       0
slice_name      Unknown
slice_code      0
slice_start     0
slice_end       0
slice_duration  0.000000
toffset         0.000000
intent          Unknown
intent_code     0
intent_name
intent_p1       0.000000
intent_p2       0.000000
intent_p3       0.000000
qform_name      Scanner Anat
qform_code      1
qto_xyz:1       0.000000 0.000000 1.000000 -89.557289
qto_xyz:2       -1.000000 0.000000 0.000000 150.847916
qto_xyz:3       0.000000 -1.000000 0.000000 120.816551
qto_xyz:4       0.000000 0.000000 0.000000 1.000000
qform_xorient   Anterior-to-Posterior
qform_yorient   Superior-to-Inferior
qform_zorient   Left-to-Right
sform_name      Unknown
sform_code      0
sto_xyz:1       0.000000 0.000000 0.000000 0.000000
sto_xyz:2       0.000000 0.000000 0.000000 0.000000
sto_xyz:3       0.000000 0.000000 0.000000 0.000000
sto_xyz:4       0.000000 0.000000 0.000000 1.000000
sform_xorient   Unknown
sform_yorient   Unknown
sform_zorient   Unknown
file_type       NIFTI-1+
file_code       1
descrip
aux_file

In this case:

sform_name Scanner Anat VS Unknown
sform_code 1 VS 0

Related issues

ivadomed/model_seg_ms_mp2rage#77

The text was updated successfully, but these errors were encountered:

jcohenadad · 2023-09-22T13:12:31Z

@Nilser3 make sure to always use appropriate labels please:

jcohenadad · 2023-09-22T13:13:20Z

To solve this problem maybe we should make sure that the q_form and the s_form of image and mask are the same

Agreed. I propose to copy the qform into the sform of all problematic derivatives data. Incl. qform_name and qform_code. @Nilser3 please take care of this

Nilser3 · 2023-09-22T14:13:48Z

Firstly I had (in the mask):

sct_image -i sub-P001_UNIT1_lesion-manualNeuroPoly.nii.gz -header

qform_name      Scanner Anat
qform_code      1
qto_xyz:1       0.000000 0.000000 1.000000 -89.557289
qto_xyz:2       -1.000000 0.000000 0.000000 150.847916
qto_xyz:3       0.000000 -1.000000 0.000000 120.816551
qto_xyz:4       0.000000 0.000000 0.000000 1.000000
qform_xorient   Anterior-to-Posterior
qform_yorient   Superior-to-Inferior
qform_zorient   Left-to-Right
sform_name      Unknown
sform_code      0
sto_xyz:1       0.000000 0.000000 0.000000 0.000000
sto_xyz:2       0.000000 0.000000 0.000000 0.000000
sto_xyz:3       0.000000 0.000000 0.000000 0.000000
sto_xyz:4       0.000000 0.000000 0.000000 1.000000
sform_xorient   Unknown
sform_yorient   Unknown
sform_zorient   Unknown

After applying:

sct_image -i sub-P001_UNIT1_lesion-manualNeuroPoly.nii.gz -set-sform-to-qform -o OUTPUT.nii.gz

sct_image -i OUTPUT.nii.gz -header

qform_name      Scanner Anat
qform_code      1
qto_xyz:1       0.000000 0.000000 1.000000 -89.557289
qto_xyz:2       -1.000000 0.000000 0.000000 150.847916
qto_xyz:3       0.000000 -1.000000 0.000000 120.816551
qto_xyz:4       0.000000 0.000000 0.000000 1.000000
qform_xorient   Anterior-to-Posterior
qform_yorient   Superior-to-Inferior
qform_zorient   Left-to-Right
sform_name      Scanner Anat
sform_code      1
sto_xyz:1       0.000000 0.000000 1.000000 -89.557289
sto_xyz:2       -1.000000 0.000000 0.000000 150.847916
sto_xyz:3       0.000000 -1.000000 0.000000 120.816551
sto_xyz:4       0.000000 0.000000 0.000000 1.000000
sform_xorient   Anterior-to-Posterior
sform_yorient   Superior-to-Inferior
sform_zorient   Left-to-Right

and finally this q_form and s_form is the same as the anatomical image.

sct_image -i sub-P001_UNIT1.nii.gz -header

qform_name      Scanner Anat
qform_code      1
qto_xyz:1       0.000000 0.000000 1.000000 -89.557289
qto_xyz:2       -1.000000 0.000000 0.000000 150.847916
qto_xyz:3       0.000000 -1.000000 0.000000 120.816551
qto_xyz:4       0.000000 0.000000 0.000000 1.000000
qform_xorient   Anterior-to-Posterior
qform_yorient   Superior-to-Inferior
qform_zorient   Left-to-Right
sform_name      Scanner Anat
sform_code      1
sto_xyz:1       -0.000000 -0.000000 1.000000 -89.557289
sto_xyz:2       -1.000000 -0.000000 -0.000000 150.847916
sto_xyz:3       0.000000 -1.000000 0.000000 120.816551
sto_xyz:4       0.000000 0.000000 0.000000 1.000000
sform_xorient   Anterior-to-Posterior
sform_yorient   Superior-to-Inferior
sform_zorient   Left-to-Right

jcohenadad · 2023-09-22T14:30:05Z

Great! Can you please

open a branch
correct all relevant images
have someone from the team review, eg @mguaypaq

Nilser3 · 2023-09-29T14:48:46Z

Hi @mguaypaq

Could I have access to basel-mp2rage like W to solve this issue and also ivadomed/model_seg_ms_mp2rage#76

mguaypaq · 2023-10-02T15:08:29Z

@Nilser3 I just gave you W access to basel-mp2rage ✔️

Nilser3 · 2023-10-02T18:07:57Z

Thanks you @mguaypaq

I have created a branch nlm/alignment_images_derivatives on basel-mp2rage dataset
where I have resolved these issues:
#265
ivadomed/model_seg_ms_mp2rage#78
ivadomed/model_seg_ms_mp2rage#76

I think I'm ready for a PR

mguaypaq · 2023-10-03T18:28:19Z

I can see your branch nlm/alignment_images_derivatives, with the filenames and folders and content hashes, but I can't see the actual images. So it looks like git push worked (from these instructions), but can you run these commands again please:

git annex copy --all --to=origin
git annex sync --no-content --only-annex

After that I'll be able to review the changes.

Nilser3 · 2023-10-03T19:26:58Z

Hi @mguaypaq
Thanks,
I just did it!

mguaypaq · 2023-10-13T17:36:25Z

Sorry for the delay, I've been busy with the upcoming SCT release.

There's a slight problem (that we can ignore) with some of the git-annexed files: for the 27 image files that were binarized in the commit 8b4b8db20df0efbc7e2893345410441a2c87bea3, we only have the most recent version of the files, not the previous version before binarization (from commit 8bdb44219483a28ef659c83bbd2c13bc1243095c). After some in-person debugging with @Nilser3, it looks like those previous versions are lost forever, and can't easily be re-generated. Oh well.

Apart from that, git annex get works fine. Since the changes only touch the derivatives/labels/ folder, bids-validator is irrelevant.

Still, I had a look at the JSON sidecars in derivatives/labels/ and noticed some problems, which I fixed in a couple of extra commits:

All of the sub-*_UNIT1_label-SC_seg.json files contained illegal JSON syntax (a trailing comma). They were all copies of the same file, so I replaced them from:
```
{
    "Author": "sct_deepseg_sc, Julien Cohen-Adad",
}
```
to:
```
{
  "Authors": [
    "sct_deepseg_sc",
    "Julien Cohen-Adad"
  ]
}
```
This fixes the problem with the trailing commas, but also changes the single Author fields to a plural Authors field with a list of strings.
All of the images modified by @Nilser3 were missing a mention of this in their JSON sidecar. Again, for these files I either created the missing JSON sidecar, with an Author of "Nilser Laines Medina", or I changed the existing single Author to an Authors list and added "Nilser Laines Medina".

After adding these fixes, I merged into master.

Since these fixes are maybe controversial, I'll let @jcohenadad close this issue as completed if he agrees.

jcohenadad · 2023-10-13T18:17:58Z

Yes, I'll have a look before merging. Thank you @mguaypaq and @Nilser3

jcohenadad · 2023-10-27T14:07:52Z

@mguaypaq I'm looking at 8b4b8db20df0efbc7e2893345410441a2c87bea3 and still see the issue with the JSON:

julien-macbook:~/data.neuro/basel-mp2rage $ cat derivatives/labels/sub-C087/anat/sub-C087_UNIT1_label-SC_seg.json
{
    "Author": "sct_deepseg_sc, Julien Cohen-Adad",
}

Did you fix it only on master or was Nilser's branch also updated with that fix? In the former case, would it be possible to update Nilser's branch so I can review the latest version. Thanks!

mguaypaq · 2023-10-27T17:08:14Z

@jcohenadad Sorry, yes, I had pushed fixed the JSON sidecars only on the master branch. But now I reset the master branch to where it was before this PR, and put the JSON sidecar fixes on the nlm/alignment_images_derivatives branch (which is now commit 114b5b276445a72237e629ed0bb810d8a43bc1d4).

You should be able to review the branch now.

I'll note that, in other datasets, we've been using the following schema for multiple authors recently:

{
  "GeneratedBy": [
    {
      "Author": "some_script",
      "Date": "2023-mm-dd HH:MM:SS"
    },
    {
      "Author": "Firstname Lastname",
      "Date": "2023-mm-dd HH:MM:SS"
    }
  ]
}

Unfortunately, in basel-mp2rage we don't have dates, and we do sometimes have a "Label" field, so it doesn't really fit the new schema.

jcohenadad · 2023-10-27T22:00:43Z

Thank you Mathieu, I will check ASAP.

Unfortunately, in basel-mp2rage we don't have dates, and we do sometimes have a "Label" field, so it doesn't really fit the new schema.

That's all right, we will do better next time we create the labels 😅

jcohenadad · 2023-11-10T15:30:07Z

I checked the latest commit 114b5b276445a72237e629ed0bb810d8a43bc1d4 and it looks good-- we can merge. Thank you @Nilser3 and @mguaypaq !

mguaypaq · 2023-11-10T15:58:08Z

Merged into master 🎉

jcohenadad added the dataset: basel-mp2rage label Sep 22, 2023

mguaypaq closed this as completed Nov 10, 2023

This was referenced Nov 10, 2023

Incorrect QC report in MS masks ivadomed/model_seg_ms_mp2rage#78

Closed

Unexpected pixels in lesion and SC segmentations ivadomed/model_seg_ms_mp2rage#76

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strategy to ensure alignment of images and derivatives #265

Strategy to ensure alignment of images and derivatives #265

Nilser3 commented Sep 21, 2023 •

edited by jcohenadad

Loading

jcohenadad commented Sep 22, 2023

jcohenadad commented Sep 22, 2023 •

edited

Loading

Nilser3 commented Sep 22, 2023

jcohenadad commented Sep 22, 2023

Nilser3 commented Sep 29, 2023

mguaypaq commented Oct 2, 2023

Nilser3 commented Oct 2, 2023

mguaypaq commented Oct 3, 2023

Nilser3 commented Oct 3, 2023 •

edited

Loading

mguaypaq commented Oct 13, 2023 •

edited

Loading

jcohenadad commented Oct 13, 2023

jcohenadad commented Oct 27, 2023

mguaypaq commented Oct 27, 2023

jcohenadad commented Oct 27, 2023

jcohenadad commented Nov 10, 2023

mguaypaq commented Nov 10, 2023

Strategy to ensure alignment of images and derivatives #265

Strategy to ensure alignment of images and derivatives #265

Comments

Nilser3 commented Sep 21, 2023 • edited by jcohenadad Loading

Description

In this case:

Related issues

jcohenadad commented Sep 22, 2023

jcohenadad commented Sep 22, 2023 • edited Loading

Nilser3 commented Sep 22, 2023

jcohenadad commented Sep 22, 2023

Nilser3 commented Sep 29, 2023

mguaypaq commented Oct 2, 2023

Nilser3 commented Oct 2, 2023

mguaypaq commented Oct 3, 2023

Nilser3 commented Oct 3, 2023 • edited Loading

mguaypaq commented Oct 13, 2023 • edited Loading

jcohenadad commented Oct 13, 2023

jcohenadad commented Oct 27, 2023

mguaypaq commented Oct 27, 2023

jcohenadad commented Oct 27, 2023

jcohenadad commented Nov 10, 2023

mguaypaq commented Nov 10, 2023

Nilser3 commented Sep 21, 2023 •

edited by jcohenadad

Loading

jcohenadad commented Sep 22, 2023 •

edited

Loading

Nilser3 commented Oct 3, 2023 •

edited

Loading

mguaypaq commented Oct 13, 2023 •

edited

Loading