WIP: add single_chunk_size_threshold opt

[magland]

@rly

I've been running into a problem where some of the large (raw ephys) datasets on DANDI are uncompressed and stored in a single very large chunk within the NWB file. For example 000935. In fact, as you know, the default in HDF5 is to create an uncompressed contiguous chunk. This is a problem for Zarr with remote files because (as far as I can see) there is no such thing as a partial read of a chunk. So if I try to randomly access a time segment with one of these arrays, the entire chunk/dataset needs to be downloaded.

This problem could potentially be addressed at various stages of the flow, but at this point I haven't found any that are satisfactory.

As a workaround, in this PR I have added another attribute to the LindiH5ZarrStoreOpts called single_uncompressed_chunk_size_threshold which defaults to 100 MB. It detects the situation where the dataset is a single uncompressed chunk of size greater than 100 MB. In this case, it is marked as an external array link. So in this case, when the client loads the array, it falls back to loading the data using h5py reader, which is capable of slicing the data efficiently.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 87.50000% with 2 lines in your changes missing coverage. Please review.

Project coverage is 79.51%. Comparing base (731fcb5) to head (bc03770).
Report is 9 commits behind head on main.

Files	Patch %	Lines
lindi/LindiH5ZarrStore/LindiH5ZarrStore.py	86.66%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #82      +/-   ##
==========================================
+ Coverage   79.43%   79.51%   +0.08%     
==========================================
  Files          30       30              
  Lines        2256     2275      +19     
==========================================
+ Hits         1792     1809      +17     
- Misses        464      466       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[magland]

Having second thoughts about this solution, so marking as WIP.

Maybe it would be better/cleaner to detect this situation and then auto split into a large number of chunks along the first dimension, so it is represented in the RFS as multiple chunks.