to_netcdf engine='h5netcdf' Chunk Dimensionality issue.

[aymanbfree]

While upgrading xarray from 2022.3.0 to 2023.8.0 we are now seeing errors:

Chunk shape must not be greater than data shape in any dimension. (1, 1, 52, 16) is not compatible with (1, 1, 52, 0)

While running ds.to_netcdf(fname, engine='h5netcdf', encoding=encoding)

I'm struggling to find documentation on what change has caused this issue so I can implement a fix. Any suggestions?

[kmuehlbauer]

@aymanbfree Please provide more context. A MCVE https://stackoverflow.com/help/minimal-reproducible-example would be great. From the information given it's hard to get to the root cause.

There have been changes to the chunking behaviour in h5netcdf. So my first question would be, if does engine="netcdf4" work for you?

[dcherian]

You have a 0 in your data shape. I would track that down

[aymanbfree]

We would need to continue using h5netcdf as its the only engine capable of reading LZF-compressed NetCDF files.

For more context of the error we are seeing:

File "/lgxarray.py", line 158, in write_netcdf_lzf
ds.to_netcdf(fname, engine='h5netcdf', encoding=encoding)
File "/anaconda/lib/python3.11/site-packages/xarray/core/dataset.py", line 2252, in to_netcdf
return to_netcdf( # type: ignore # mypy cannot resolve the overloads:(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/anaconda/lib/python3.11/site-packages/xarray/backends/api.py", line 1246, in to_netcdf
dump_to_store(
File "/anaconda/lib/python3.11/site-packages/xarray/backends/api.py", line 1293, in dump_to_store
store.store(variables, attrs, check_encoding, writer, unlimited_dims=unlimited_dims)
File "/anaconda/lib/python3.11/site-packages/xarray/backends/common.py", line 356, in store
self.set_variables(
File "/anaconda/lib/python3.11/site-packages/xarray/backends/common.py", line 394, in set_variables
target, source = self.prepare_variable(
^^^^^^^^^^^^^^^^^^^^^^
File "/anaconda/lib/python3.11/site-packages/xarray/backends/h5netcdf_.py", line 316, in prepare_variable
nc4_var = self.ds.create_variable(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/anaconda/lib/python3.11/site-packages/h5netcdf/core.py", line 869, in create_variable
return group._create_child_variable(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/anaconda/lib/python3.11/site-packages/h5netcdf/core.py", line 736, in _create_child_variable
self._h5group.create_dataset(
File "/anaconda/lib/python3.11/site-packages/h5py/_hl/group.py", line 183, in create_dataset
dsid = dataset.make_new_dset(group, shape, dtype, data, name, **kwds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/anaconda/lib/python3.11/site-packages/h5py/_hl/dataset.py", line 72, in make_new_dset
raise ValueError(errmsg)

CRITICAL root:batch.py:525 Chunk shape must not be greater than data shape in any dimension. (1, 1, 52, 16) is not compatible with (1, 1, 52, 0)
CRITICAL root:batch.py:526 FAILED Batch: MtFCache - With 0 critical errors, 0 errors, and 0 warnings

And some more context of the code snippet that we find the issue:

def write_netcdf_lzf(ds, fname, *, chunks=True):
"""Write out a :class:xarray.Dataset to NetCDF format, using LZF
compression on all variables.

The output file can be read back with::

    xarray.open_dataset(fname, engine='h5netcdf')

:param ds:
    xarray.Dataset
:param fname:
    output NetCDF file name to write to
:param chunks:
    dict of {dim: chunk size}
    Unlike the rest of xarray and dask, does not support tuple notation.
    Set to True to let h5py do auto-chunking.
"""
fname = sh.resolve_env(fname)

encoding = {}
for k, v in ds.variables.items():
    encoding[k] = {
        # Enable checksums against corrupted data.
        # No adverse impact on performance or size.
        'fletcher32': True,
        # Enable compression with the LZF algorithm
        'compression': 'lzf',
        # Improve compression rate by internally reordering samples
        'shuffle': True,
    }

    if chunks is True:
        encoding[k]['chunksizes'] = True
    elif isinstance(chunks, dict):
        encoding[k]['chunksizes'] = tuple(
            min(size, chunks.get(dim, size))
            for dim, size in v.sizes.items())
    else:
        raise TypeError("chunks must be true or a dict {dim: chunk size}")

    if v.dtype.kind == 'U':
        # Drastically improve strings compression ratio.
        # See:
        # - http://xarray.pydata.org/en/stable/io.html#string-encoding
        # - https://github.com/pydata/xarray/issues/2040
        encoding[k]['dtype'] = 'S1'
        encoding[k]['chunksizes'] = True
        pass
ds.to_netcdf(fname, engine='h5netcdf', encoding=encoding)

[kmuehlbauer]

Thanks for the additional information. Please check why your data shape contains one zero-sized dimension, as @dcherian suggested.