LocalCache: limit allowed blob size

[magland]

I found a situation where the chunk size was larger than 1 GB and the sqllite save failed. This particular dataset was not chunked, and this should not be an issue for most dandisets where chunk sizes are reasonable.

So with this PR I limit the size of chunks in local cache to 900 MB.

See https://www.sqlite.org/limits.html

(For some reason, despite what it says in that limits.html, it seems that 1000 MB doesn't work whereas 900 MB does)

For LindiH5ZarrStore and LindiReferenceFileSystemStore, I display a warning if the chunk size is too large, and then skip storing it in local cache. For LindiRemfile I raise an exception because this should never happen in that context.

I added a test.

[rly]

Since the 900 MB limit is specific to the sqlite cache and I would like to keep the magic numbers in one place rather than four, could you move the if statement to LocalCacheSQLiteClient.put_remote_chunk and raise an exception there if it is too big? We can catch the exception and raise a warning instead in LindiH5ZarrStore and LindiReferenceFileSystemStore.

I'm also happy to refactor that.

[magland]

Since the 900 MB limit is specific to the sqlite cache and I would like to keep the magic numbers in one place rather than four, could you move the if statement to LocalCacheSQLiteClient.put_remote_chunk and raise an exception there if it is too big? We can catch the exception and raise a warning instead in LindiH5ZarrStore and LindiReferenceFileSystemStore.

I'm also happy to refactor that.

Sounds good. I'll work on that.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 50.00000% with 8 lines in your changes are missing coverage. Please review.

Project coverage is 79.41%. Comparing base (797869d) to head (1535202).

Files	Patch %	Lines
lindi/LindiH5ZarrStore/LindiH5ZarrStore.py	20.00%	4 Missing ⚠️
...ndi/LindiH5pyFile/LindiReferenceFileSystemStore.py	20.00%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #69      +/-   ##
==========================================
- Coverage   79.58%   79.41%   -0.17%     
==========================================
  Files          30       30              
  Lines        2243     2254      +11     
==========================================
+ Hits         1785     1790       +5     
- Misses        458      464       +6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[magland]

@rly Is this what you had in mind?

[rly]

Yes, awesome. Thanks for the change!

[magland]

on pypi as 0.3.7