How to truncate an electrical series and preserve compression?

[DailyDreaming]

I have several NWB files that have an electrical series where I'd like to trim off the first noisy 70 seconds or so. What is the best way of doing this?

The only reliable way I've found so far is using h5py and reshaping the dataset:

with h5py.File("original.raw.h5.nwb", "r+") as f:
    print(f["acquisition"]['ElectricalSeries']['data'].compression)  # gzip
    data = f["acquisition"]['ElectricalSeries']['data'][1400000:, :952]
    new_io = H5DataIO(data, compression=True, link_data=False, chunks=True)
    f["acquisition"]['ElectricalSeries'].pop('data')
    f["acquisition"]['ElectricalSeries']['data'] = new_io

with h5py.File("original.raw.h5.nwb", "r+") as f:
    print(f["acquisition"]['ElectricalSeries']['data'].compression)  # None

However, the resulting dataset results in a file that was ~13Gb ballooning to ~30Gb and when inspected, the dataset is uncompressed (compression=None). Is there a way modify the dataset to still have gzip compression?

Thanks in advance!

[bendichter]

Yes, in HDF5 if you delete an old dataset and create a new one, the old dataset still takes up space. What you should do instead is:

with h5py.File("original.raw.h5.nwb", "r+") as f:
    trimmed_data = f["acquisition"]['ElectricalSeries']['data'][1400000:, :952]
    f["acquisition"]['ElectricalSeries']['data'][:-1400000, :952] = trimmed_data
    f["acquisition"]['ElectricalSeries']['data'].resize((1400000, 952))

This will only work if the dataset is chunked to begin with (which is true in your case)

[bendichter]

Make sure to update your timestamps or starting_time accordingly!

[DailyDreaming]

Thank you!