Add support for DICOM dermoscopy objects

[fedorov]

This follows up on the discussion with @Punzo on slack.

The samples can be accessed in https://challenge2020.isic-archive.com/.

If it turns out the effort required is significant, we will need to reconsider.

FYI: @dclunie

[Punzo]

@fedorov it turns out that OHIFv2 can load dermoscopy data without any issue.

I loaded locally the DICOM corrected Test Data (6,7 GB) from https://challenge2020.isic-archive.com/ .

There is an issue: in the metadata the PhotometricInterpretation is set to YBR_FULL_422, however when downloading the image the array length is = rows * columns * 3 (instead of 2, i.e., each pixel should have 2 values in YBR_FULL_422, which can be converted to RGB/RGBA values, see link).

if I force the PhotometricInterpretation to RGB, the image is loaded properly:

Therefore, I think that the images are actually encoded in RGB. I could put a check on the downloaded image array and set a special case (i.e., if the data array length is = rows * columns * 3 and PhotometricInterpretation = YBR_FULL_422, ignore PhotometricInterpretation and convert anyway as RGB images) in https://github.com/cornerstonejs/cornerstoneWADOImageLoader/blob/master/src/imageLoader/convertColorSpace.js, but I would prefer to avoid putting hacks in place for defective datasets.

3DSlicer loads the images without problems, but I think it ignores the PhotometricInterpretation, and it just uses the SamplePerPixel attribute (which for YBR_FULL_422, SamplePerPixel is 3 by the standard, even if before the conversion to RGB/RGBA the array values for each pixel are 2).

Anyway, for datasets with non-defective metadata, DICOM dermoscopy objecjs will be loaded with no problems with OHIFv2 with latests cornerstoneWADOImageLoader (see PR: #2944)

[fedorov]

I would prefer to avoid putting hacks in place for defective datasets

I agree.

I believe @dclunie is quite familiar with this dataset - would be good to have his guidance here.

[dclunie]

@Punzo, AFAIK, all these images are JPEG baseline compressed (Transfer Syntax 1.2.840.10008.1.2.4.50).

The DICOM Photometric Interpretation describes what is encoded in the compressed JPEG bitstream, not the RGB that might (or might not) come out of a JPEG decoder that converts the YCbCr chrominance downsampled compressed data into uncompressed RGB. See:

https://dicom.nema.org/medical/dicom/current/output/chtml/part05/sect_8.2.html#para_4bcb841e-c6bf-4e26-82a5-3fad3c942da0

and the notes after the table.

So, the logic in OHIF (or the image loader it is using) needs to account for whatever the JPEG decoder it is using actually does, and not be literally applied to the decompressed pixel data (i.e., it should know there are 3 pixels per sample in the decompressed data not 2 if chrominance downsampling has already been reversed), and that the color space has already been transformed by the decoder from YCbCr to RGB (both are usually the case for baseline JPEG decoders such as those that are libpeg-based).

Or to put it another way, each DECOMPRESSED pixel should have 3 values not 2 as you are assuming, regardless of the PhotometricInterpretation of the COMPRESSED pixel data being YBR_FULL_422. The only time that the PixelData length should be considered in this respect is when the encoding is "Native" (uncompressed), not "Encapsulated" (compressed), i.e., Note 2:

https://dicom.nema.org/medical/dicom/current/output/chtml/part03/sect_C.7.6.3.html#para_442b66ec-6d91-42ba-bfeb-738c94d2d6d3

does not apply to these JPEG compressed images.

Your decoder should not be guessing that Photometric Interpretation might be "wrong", rather it should know exactly what the compressed pixel data decoder has done, and behave accordingly.

Be aware that not all JPEG compressed pixel data is YCbCr - sometimes it really was compressed as RGB components (e.g., Aperio WSI).

Also, for JPEG compressed images, though YBR_FULL_422 is always used when a YCbCr color transformation has been applied before compression, the actual extent of chrominance downsampling varies in practice, and is not always 2x2:

https://dicom.nema.org/medical/dicom/current/output/chtml/part03/sect_C.7.6.3.html#para_054ed567-c63e-4450-a128-0c8af7adc764

Further, some of the YBR family of Photometric Interpretations are used for non-JPEG compressed pixel data (e.g., some RLE or uncompressed ultrasound images, in which case a color space transformation +/- a chrominance channel upsampling may be required). That should not apply to these ISIC dermatology challenge images, though changes to your logic might affect handling of other images, which is why I mention it.

@chafey might want to comment on what Cornerstone does/should do in this respect for compressed images.

[Punzo]

Found the issue. The wado parser in cornerstoneWADOImageLoader sends a XMLHttpRequest as type=application/octet-stream; transfer-syntax=*
Then it uses the mime information from the server response to guess the transfer syntax of the donwloaded frame.

https://github.com/cornerstonejs/cornerstoneWADOImageLoader/blob/master/src/imageLoader/wadors/loadImage.js#L79
https://github.com/cornerstonejs/cornerstoneWADOImageLoader/blob/master/src/imageLoader/wadors/loadImage.js#L10

However in the case of requests as type=application/octet-stream; transfer-syntax=* the right mime will not be known
(we had similar issues with SLIM: https://github.com/herrmannlab/dicom-microscopy-viewer/blob/master/src/pyramid.js#L426-L433). In fact in the case of this dermoscopy datasets, we would not get an a "image/jpeg" mime response (i.e. the frame is decompressed by the server and returned already as uncompressed color converted rgb image).

Therefore the cornerstoneWADOImageLoader will load internally the frame as if the Transfer Syntax is 1.2.840.10008.1.2 instead of 1.2.840.10008.1.2.4.50. This finallly generates issues, because color transformations are applied client side since the guessed transfer syntax is wrong (not all decoders apply automatically the color transformations, so for a set of transfer syntaxes corresponding to such decoders, the color treansformations are appllied as in https://github.com/cornerstonejs/cornerstoneWADOImageLoader/blob/master/src/imageLoader/createImage.js#L238).

We could improve the cornerstoneWADOImageLoader either:

1. using the dicom-webclient instead to use a simple XMLHttpRequest (e.g., in Slim we use it to pass an array of transfer syntaxes https://github.com/herrmannlab/dicom-microscopy-viewer/blob/83739f1ba11840df7f4487cdf8015b7450ee419d/src/pyramid.js#L436-L491 and the first compatible one will used). In this way (having the "image/jpeg" mime of higher priority than application/octet-stream), we can get the original compressed jpeg file for the frame and we will decompress/apply color transformations client side.

2. After fetching the arrayBuffer, we could guess the right frame mime from the array buffer rather then from the mime information from the server response. E.g. similar to the approach that we use in SLIM: https://github.com/herrmannlab/dicom-microscopy-viewer/blob/83739f1ba11840df7f4487cdf8015b7450ee419d/src/webWorker/decodeAndTransformTask.js#L99-L150

@sedghi @swederik, it would be nice to have your feedback as well. Thanks!

[Punzo]

@fedorov moving this back in Progress

[Punzo]

open PR with fix at cornerstonejs/cornerstoneWADOImageLoader#477

[fedorov]

Per @igoroctaviano, could it be related to #3354 ?

[jlulloaa]

A latest update to ohif (v3-stable) contains updates to cornerstone that solved the issue with the RGB display. Cornerstone's CPU-based thumbnails rendering didn't interpret correctly the RGB dermoscopy images, affecting how the thumbnails were displayed in OHIF. It was solved by changing the behaviour to GPU rendering by default (cornerstonejs/cornerstone3D#586). This feature was transferred to OHIF in PR 3372 and now available in v3-stable.

This was tested with a subset from "DICOM corrected Test Data (6,7 GB)", available at (https://challenge2020.isic-archive.com/)
OHIF/Viewer (v3-stable, Cornerstone Tools v0.66.2):

OHIF/Viewer (v3-stable, Cornerstone Tools v0.67.1):

Same data loaded through https://v3-demo.ohif.org

[jlulloaa]

Per @igoroctaviano, could it be related to #3354 ?

Doesn't seem to be the same. It is not fixed with the update to cornerstone tools. From the data, it looks like the issue may be related to the way ohif viewer decode the RGB overlaid mask (I'll add more details about that directly in #3354). Also, in #3354 the thumbnails look ok, but not the images in the viewer, here instead, the images looked wrong in both, thumbnail and main viewer

[jlulloaa]

A latest update to ohif (v3-stable) contains updates to cornerstone that solved the issue with the RGB display. Cornerstone's CPU-based thumbnails rendering didn't interpret correctly the RGB dermoscopy images, affecting how the thumbnails were displayed in OHIF. It was solved by changing the behaviour to GPU rendering by default (cornerstonejs/cornerstone3D-beta#586). This feature was transferred to OHIF in PR 3372 and now available in v3-stable.

This was tested with a subset from "DICOM corrected Test Data (6,7 GB)", available at (https://challenge2020.isic-archive.com/) OHIF/Viewer (v3-stable, Cornerstone Tools v0.66.2):

OHIF/Viewer (v3-stable, Cornerstone Tools v0.67.1):

Same data loaded through https://v3-demo.ohif.org

As shown in this test, this issue is solved with the latest updates to v3-stable and the data is now displayed correctly (https://v3-demo.ohif.org/).

I think this issue can be closed, @fedorov @igoroctaviano

[fedorov]

There is an unresolved known issue related to v3 incorrectly handling multichannel images: #3354.

I guess in the context of this issue, the question is: why is it that dermoscopy images render fine, and other RGB examples in #3354 do not?

[vicongit23]

My examples are also US ( as in 3354). Also PALETTE _COLOR doesn't work.
basic_viewer

[vicongit23]

IMO dermoscopy images ( at least seen in OHIF v3 worklist) have photometric interpretation YBR_ICT

[rodrigobasilio2022]

I need to see the image to understand the problem. If you can upload to the dicom store ( I dont have the autority to let you), I can take a look in tje image.

[vicongit23]

Please look at 3354,
sample 3 displays the same presentation on similar US modality file

[rodrigobasilio2022]

You are from IDC? I don't have full access so I need the url to the image so I can access it

[fedorov]

No, I do not think @vicongit23 is from IDC. As suggested above, let's continue the discussion in #3354. Of note, there are links to the public samples from IDC in that issue.

[vicongit23]

No, I do not think @vicongit23 is from IDC. As suggested above, let's continue the discussion in #3354. Of note, there are links to the public samples from IDC in that issue.

Correct

[fedorov]

This was broken https://d3w0jbfdn2j3pj.cloudfront.net/projects/idc-sandbox-000/locations/us-central1/datasets/isic/dicomStores/isic2020_test/study/1.3.6.1.4.1.5962.99.1.1195.6460.1620398329167.1.2.1

[igoroctaviano]

This was broken https://d3w0jbfdn2j3pj.cloudfront.net/projects/idc-sandbox-000/locations/us-central1/datasets/isic/dicomStores/isic2020_test/study/1.3.6.1.4.1.5962.99.1.1195.6460.1620398329167.1.2.1

I opened this one today and it looked good, at least the first series. There's a specific broken series?

[igoroctaviano]

I tried again with master and I got different results this time, it's still broken:

[igoroctaviano]

I tried again with master and I got different results this time, it's still broken:

@sedghi FYI, looks like a regression

[igoroctaviano]

@sedghi do we have automation in place for the codec library? running snapshot testing and checking if the images change? I think this is very useful to avoid regression.

[sedghi]

sad :(
No we don't have really but we should really. Can you give me that data in .dcm format?

[igoroctaviano]

sad :( No we don't have really but we should really. Can you give me that data in .dcm format?

@fedorov is it ok to share this dataset?

[fedorov]

Yes! But note that it will be prudent for you to keep track of where a given test dataset is coming from, and make sure you are in compliance with the license. This dermoscopy sample is from ISIC: https://challenge2020.isic-archive.com/

[igoroctaviano]

sad :( No we don't have really but we should really. Can you give me that data in .dcm format?

@sedghi I uploaded the isic dataset in the same drive I shared with you previously with other datasets.

[fedorov]

We do not seem to have any images in IDC that would be suitable to reproduce this issue.