Add hdf5

[mverleg]

No description provided.

[mverleg]

I'm skeptical that this measures real write speed, because it's faster than binary.

I have a flush in there, I don't know what else could be done.

[gaopinghai]

will h5 file have problem with CPU memory？I encounter some problem which I think is caused by the memory consumption of loading h5file，especially when using multi-workers

[mverleg]

@PingHGao I think this might not be the best place to get help with your problem sorry

[Nimrod0901]

I'm skeptical that this measures real write speed, because it's faster than binary.

I have a flush in there, I don't know what else could be done.

Hi, is this related to chunk caching of HDF5? If you're writing or reading the same content(chunk) multiple times, real I/O will be executed once.

If you're using h5py, you can set the rdcc_* arguments when opening a file to disable the cache effect. https://docs.h5py.org/en/stable/high/file.html#chunk-cache