[Bug]: Pixel_mask doesn't seem to be properly constructed.

[chrisvdt]

What happened?

Although my nwb object does get exported to an NWB file without errors, and the validation with python is successfull, I cannot view the Imagesegmentation in neurosift.

A raw view shows that the pixel_map is not a compound datatype object array.

I made this structure array with struct('x', uint32(0), 'y', uint32(0), 'weight', single(0)). But I see that matnwb writeCompound converts them all to doubles.

Steps to Reproduce

mask = struct('x', uint32(0), 'y', uint32(0), 'weight', single(0));

                       mask_index = uint8(zeros(nRois, 1));
                       bi = 0;
                       for i = 1:nRois
                           [y, x] = find(ROI_data.Mask == i);
                           ln = length(x);
                                                     
                           mask_index(i) = ln;
                          
                           for j = 1:ln
                                mask(bi + j).x = x(j);
                                mask(bi + j).y = y(j);
                                mask(bi + j).weight = 1.0;
                           end
                           bi = bi + ln;
                       end

                       pix_mask = types.hdmf_common.VectorData('data', mask, 'description', 'roi pixel position (x,y) and pixel weight');

Error Message

No response

Operating System

Windows

Matlab Version

matlab R2021a

Code of Conduct

• I agree to follow this project's Code of Conduct
• Have you ensured this bug was not already reported?

[chrisvdt]

I wrote earlier that writeCompound converted all the fields to doubles. That is not true, my structure arrays were not formatted properly. I changed my code and now the structure arrays keep their original data types uint32, uint32, single.
The result is however the same, pixel_mask is an {Object Object] array, and neurosift does not display the pixel_mask.

[bendichter]

What it looks like for a file generated with PyNWB:

[ehennestad]

In order to create a proper compound type for pixel_masks you could do the following (see code below). Note: this is one way I found that worked, and this process should be documented better for matnwb. Another note; as I do not have your ROI_data structure, I have not run the code to verify that it works. But the key is to use a struct where each field is a vector of all the concatenated values for x, y and weight respectively, and to also specify the pixel_mask_idx to create a ragged array

A second note: In order to visualise these masks in neurosift you need to add an image (i.e a fov projection image) in the ophys processing module. This is because there is no information about the image size in the pixel mask data, and neurosift needs to know the image size in order to reconstruct the masks in a plot. Neurosift does that by looking for an Image object in the processing module adjacent to where the ImageSegmentation object is located. See the second code snippet below for an example.

A final note: I could not reproduce exactly the view that Ben posted above for the pixel_mask property when looking at the raw view in neurosift, even though the code below should organise the data in the NWB file in the exact same way as the example file from Ben's screenshot.

Create a plane segmentation with pixel masks

imaging_plane = []; % Should be replaced with an actual image plane object (Used below).

% Initialize data
[x, y] = deal([]);
num_pixels_per_roi = zeros(nRois, 1); % Column vector

% Collect x and y indices for all rois in a column vector
for i = 1:nRois
    [iy, ix] = find(ROI_data.Mask == i);
    x = cat(1, x, ix(:)'); % Create a column vector
    y = cat(1, y, iy(:)');
    num_pixels_per_roi(i) = numel(ix);
end

% Create a struct
pixel_mask = struct;
pixel_mask.x = uint32(x);
pixel_mask.y = uint32(y);
pixel_mask.weigth = single(ones(size(x)));

% Create pixel_mask_idx vector
pixel_mask_idx = types.hdmf_common.VectorIndex(...
        'description', 'Index into pixel_mask VectorData', ...
        'data', cumsum(num_pixels_per_roi), ...
        'target', types.untyped.ObjectView('/processing/ophys/ImageSegmentation/PlaneSegmentation/pixel_mask') );

plane_segmentation = types.core.PlaneSegmentation( ...
    'colnames', {'pixel_mask'}, ...
    'description', 'roi pixel position (x,y) and pixel weight', ...
    'id', types.hdmf_common.ElementIdentifiers('data', int64(0:nRois-1)'), ...
    'imaging_plane', types.untyped.SoftLink(imaging_plane), ...
    'pixel_mask_index', pixel_mask_idx, ...
    'pixel_mask', types.hdmf_common.VectorData('data',struct2table(pixel_mask), 'description', 'pixel masks') ...
);
% When adding the compound pixel_mask to data struct2table is used. This was based on another example in the convertTrials tutorial.

Adding a mock projection image so neurosift can resolve the image (fov) size

imgs = types.core.Images('description', 'projections')
img = types.core.Image('description', 'projection', 'data', zeros(512,512))
imgs.image.set('projection_image', img)
ophys_module.nwbdatainterface.set('ImageProjection', imgs);

@chrisvdt Let me know if you have a chance to test this and if it works for you or not!

[bendichter]

nice work, @ehennestad ! Can you copy/paste a screenshot of the neurosift raw view?

[chrisvdt]

@ehennestad Many thanks, it works now. The crucial difference was this line: 'data', cumsum(num_pixels_per_roi), ...
I was simply putting an array with the num_pixels_per_roi in the data field for the pixel_mask_index. It should be clearly stated in the turorials that this needs to be a cummulative sum. (but maybe I've simply missed that information).

I tried various ways of creating a struct for the x,y and weight values. And ended up making a struct array. But this really made no difference, since the matnwb script writeCompound, converts the struct array to a single struct with three arrays like you demonstrated.

[chrisvdt]

The extra image is also important, I was wondering how neurosift could estimate the dimensions of the actual mask. My raw view however is precisely the same.

[bendichter]

Hmm, strange that the pixel mask still shows up as "object Object". @ehennestad could you take a look in HDFView?

[ehennestad]

I have compared the example file on dandiarchive with one I generated using matnwb. It is notable that the pixel_mask_index also have a different raw view for files generated in matnwb vs the file on dandi.

I have inspected / compared the two files using both matnwb and h5disp and the pixel_mask datasets appears identical. There were some differences in pixel_mask_index. I will look into how neurosift generates the view, maybe that can give some hints why the raw view is different

[bendichter]

fixed by #577