[ENH] Extend microscopy spec to support NGFF

[satra]

As we (in the dandiarchive) are beginning to distribute substantial amount (320TB and growing) microscopy data, some of which are in NGFF format (https://dandiarchive.org/dandiset/000108), we believe it may be time to extend the spec to support NGFF (https://www.nature.com/articles/s41592-021-01326-w, https://ngff.openmicroscopy.org/latest/).

NGFF is a zarr based format, and hence it is a directory rather than file, and it is a cloud native format to support really large datasets, which a lot of microscopy data will be for the human brain. This format thus allows direct access to the data as a data stream, without downloading all of it. More generally these datasets are unlikely to be ever moved out of the cloud at that scale, and thus this format, which includes multi-scale representation, allows any downstream user to visualize or compute on different scales of data.

cc/ @bids-standard/bep031

[Remi-Gau]

I suspect this will require a schema update:

bids-specification/src/schema/rules/datatypes/micr.yaml

Line 21 in b54ab48

extensions:

[tsalo]

It will also need a definition within objects/extensions.yaml:

bids-specification/src/schema/objects/extensions.yaml

Lines 136 to 146 in b54ab48

	.mrk:
	name: MRK
	description: |
	A file containing MEG sensor coil positions.
	Used by KIT, Yokogawa, and Ricoh MEG systems.
	Successor to the `.sqd` extension for marker files.
	.nii:
	name: NIfTI
	description: |
	A Neuroimaging Informatics Technology Initiative (NIfTI) data file.

[TheChymera]

@satra unless you already have this ready or think you can more efficiently handle this personaly, I can add the extra requested info in a new PR.

There was however a mention that this should be accompanied by example NGFF dummy data: #881 (comment)

We already have something that might provide that, which is a modified testdata set by me, here: https://github.com/dandi/bids-examples/tree/master/micr_SEM-dandi/sub-01/ses-01/micr/sub-01_ses-01_sample-A_SPIM.ngff

But it's just a bash-generated emulation of what ngff directories kind-of look like.
Not sure whether that's good enough.
@tsalo should I submit a PR with this to bids-examples?

tree output (folding added by @yarikoptic)

chymera@decohost ~/src/bids-examples $ tree ./micr_SEM-dandi/sub-01/ses-01/micr/sub-01_ses-01_sample-A_SPIM.ngff
./micr_SEM-dandi/sub-01/ses-01/micr/sub-01_ses-01_sample-A_SPIM.ngff
├── 01
│   └── 0
│       └── 0
├── 02
│   └── 0
│       └── 0
├── 03
│   └── 0
│       └── 0
├── 04
│   └── 0
│       └── 0
├── 05
│   └── 0
│       └── 0
├── 06
│   └── 0
│       └── 0
├── 07
│   └── 0
│       └── 0
├── 08
│   └── 0
│       └── 0
├── 09
│   └── 0
│       └── 0
├── 10
│   └── 0
│       └── 0
├── 11
│   └── 0
│       └── 0
├── 12
│   └── 0
│       └── 0
├── 13
│   └── 0
│       └── 0
├── 14
│   └── 0
│       └── 0
├── 15
│   └── 0
│       └── 0
├── 16
│   └── 0
│       └── 0
├── 17
│   └── 0
│       └── 0
├── 18
│   └── 0
│       └── 0
├── 19
│   └── 0
│       └── 0
├── 20
│   └── 0
│       └── 0
├── 21
│   └── 0
│       └── 0
├── 22
│   └── 0
│       └── 0
├── 23
│   └── 0
│       └── 0
├── 24
│   └── 0
│       └── 0
├── 25
│   └── 0
│       └── 0
├── 26
│   └── 0
│       └── 0
├── 27
│   └── 0
│       └── 0
├── 28
│   └── 0
│       └── 0
├── 29
│   └── 0
│       └── 0
├── 30
│   └── 0
│       └── 0
├── 31
│   └── 0
│       └── 0
├── 32
│   └── 0
│       └── 0
├── 33
│   └── 0
│       └── 0
├── 34
│   └── 0
│       └── 0
├── 35
│   └── 0
│       └── 0
├── 36
│   └── 0
│       └── 0
├── 37
│   └── 0
│       └── 0
├── 38
│   └── 0
│       └── 0
└── 39
    └── 0
        └── 0

Lastly, should we maybe use this opportunity to add .h5 support as well? It's the other microscopy format we have seen quite a lot of in DANDI and which is not yet considered valid by vanilla BIDS.

[satra]

nless you already have this ready or think you can more efficiently handle this personaly, I can add the extra requested info in a new PR.

@TheChymera - yes please, thank you. feel free to add to this PR.

with this gist (https://gist.github.com/satra/28b797fa858ff5c603d77f1cc2241098), we could create any example ngff dataset. it may be useful to have both a multiscale one and a non-multiscale one.

the h5 files were temporary, and i don't necessarily see ourselves going back to it at. so let's not worry about that.

[yarikoptic]

not sure why @TheChymera created a new PR instead of contributing to this (might need to be added to the team?), but I think #1104 supersedes this one, so I will close in that PR favor.

[TheChymera]

@yarikoptic didn't have push access to this :(