ttest2D_cr0.h5 fails trying to append to file which does not yet exist

[drew-parsons]

Issue ported from https://bitbucket.org/mpi4py/mpi4py-fft/issues/21

I'm trying to run the tests for mpi4py-fft 2.0.4 with h5py 3.3.0

Tests initially run successfully, but then fail:

$ H5PY_ALWAYS_USE_MPI=1 ./runtests.sh 
/usr/lib/python3/dist-packages/mpi4py_fft/libfft.py:182: ComplexWarning: Casting complex values to real discards the imaginary part
  self.output_array[...] = self.xfftn(self.input_array, **self.opt)
backend:  scipy 0.07071471214294434
backend:  numpy 0.06495881080627441
backend:  fftw 0.014360666275024414
Traceback (most recent call last):
  File "/home/drew/projects/python/build/mpi4py-fft/tests/test_io.py", line 181, in <module>
    test_2D(bnd, kind)
  File "/home/drew/projects/python/build/mpi4py-fft/tests/test_io.py", line 46, in test_2D
    u.write('t'+filename, 'u', 0)
  File "/usr/lib/python3/dist-packages/mpi4py_fft/distarray.py", line 400, in write
    f = writer(filename, domain=domain, mode='a')
  File "/usr/lib/python3/dist-packages/mpi4py_fft/io/h5py_file.py", line 33, in __init__
    self.f = h5py.File(h5name, mode, driver="mpio", comm=comm, **kw)
  File "/usr/lib/python3/dist-packages/h5py/_debian_h5py_mpi/_hl/files.py", line 444, in __init__
    fid = make_fid(name, mode, userblock_size,
  File "/usr/lib/python3/dist-packages/h5py/_debian_h5py_mpi/_hl/files.py", line 211, in make_fid
    fid = h5f.open(name, h5f.ACC_RDWR, fapl=fapl)
  File "h5py/_debian_h5py_mpi/_objects.pyx", line 54, in h5py._debian_h5py_mpi._objects.with_phil.wrapper
  File "h5py/_debian_h5py_mpi/_objects.pyx", line 55, in h5py._debian_h5py_mpi._objects.with_phil.wrapper
  File "h5py/_debian_h5py_mpi/h5f.pyx", line 100, in h5py._debian_h5py_mpi.h5f.open
OSError: Unable to open file (MPI_ERR_NO_SUCH_FILE: no such file or directory)

‌

Tracing the value of filename shows that this error occurs with ttest2D_cr0.h5, accessing with mode 'a'.

Preceding tests completed succcessfully (e.g.uvtest_cr1.h5, uvtest_rr1.h5, uvtest_rr2.h5). Tracing their file access conditions, I see that they were first accessed with mode 'w', and later with mode 'a'.

The error seems to be happening since ttest2D_cr0.h5 is immediately being accessed with mode 'a' without first creating the file with mode 'w'. Evidentally mode 'a' is failing since the file does not exist yet.

[drew-parsons]

@mikaem For convenience, I'll copy Mikael's comments from Issue #21.

Mikael Mortensen

I can reproduce this. The 'a' (append) mode is default when opening a file for the first time (see here), because it should create the file if it does not exist. At least according to the documentation. Will have a closer look.

There’s no error with h5py version 2.10.0. From what I can see the default used by h5py is now to open a file in read-only mode. That changed in version 3. But I don’t understand why this should affect us, since we specify mode='a' specifically. But clearly, in version 3 the 'a' mode leads to an error if the file does not exist.

Seems like h5py version 3.2.1 worked just fine as well .

[drew-parsons]

This may be a known bug in h5py 3.3 fixed in 3.4. From the h5py 3.4 changelog,

Fix bug introduced in version 3.3 that did not allow the creation of files
  using the flag "a" for certain drivers (e.g. mpiio, core and stdio)
  [PR #1922](https://github.com/h5py/h5py/pull/1922)