VisIt engine crashes on plotfiles produced with async_out=1

[BenWibking]

I'm visualizing some plotfile outputs produced by my code. I've used amrex.async_out=1 on the command line and the outputs are indeed written asynchronously. However, the VisIt engine crashes unpredictably when reading such files. For instance, I get error messages like this:

BoxLib::Error::FABio::read_header(): expected 'F' !!!
BoxLib::Error::FABio::read_header(): expected 'F' !!!
BoxLib::Error::FABio::read_header(): expected 'F' !!!
BoxLib::Error::FABio::read_header(): expected 'F' !!!
BoxLib::Error::FABio::read_header(): expected 'F' !!!
BoxLib::Error::FABio::read_header(): expected 'F' !!!

Shortly thereafter, the following occured...

The compute engine running on avatargpue.anu.edu.au has exited abnormally.

So far, I've only seen this crash happen when I've enabled async outputs.

The simulation process has finished successfully before I try to read in the plotfiles. I am reading them in from the same node that wrote them, but both reading/writing over NFS (unfortunately). Is this likely to be an NFS consistency issue, or a bug in VisIt, or something else?

[drummerdoc]

Can you grep the _H files in the plotfile for nan's?

[BenWibking]

Yes, and there are no NaNs in the *_H files.

[WeiqunZhang]

How big is the plotfile? Could you send me a copy? Either the meta-data in the header or the floating pointer data themselves are wrong.

[BenWibking]

They can be easily uploaded to cloud storage for debugging purposes. However, I deleted the previous plotfiles and unfortunately, he error doesn't seem to be deterministic. I will try again tomorrow to see if I can reproduce this.

[BenWibking]

I also occasionally get errors when writing plotfiles (but only when async_out=1, but not every time) like this:

Coarse STEP 97 at t = 0.0173205901 starts ...
Writing plotfile plt00097
terminate called after throwing an instance of 'std::runtime_error'
  what():  Couldn't open file: plt00096/Level_0/Cell_D_00012
[avatargpue:1743864] *** Process received signal ***
[avatargpue:1743864] Signal: Aborted (6)
[avatargpue:1743864] Signal code:  (-6)
[avatargpue:1743864] [ 0] /lib64/libpthread.so.0(+0x12b20)[0x7f34a9158b20]
[avatargpue:1743864] [ 1] /lib64/libc.so.6(gsignal+0x10f)[0x7f34a8db837f]
[avatargpue:1743864] [ 2] /lib64/libc.so.6(abort+0x127)[0x7f34a8da2db5]
[avatargpue:1743864] [ 3] /lib64/libstdc++.so.6(+0x9009b)[0x7f34a999009b]
[avatargpue:1743864] [ 4] /lib64/libstdc++.so.6(+0x9653c)[0x7f34a999653c]
[avatargpue:1743864] [ 5] /lib64/libstdc++.so.6(+0x96597)[0x7f34a9996597]
[avatargpue:1743864] [ 6] /lib64/libstdc++.so.6(+0x967f8)[0x7f34a99967f8]
[avatargpue:1743864] [ 7] ../build/src/test_hydro2d_kh[0x435278]
[avatargpue:1743864] [ 8] ../build/src/test_hydro2d_kh[0x4697ee]
[avatargpue:1743864] [ 9] ../build/src/test_hydro2d_kh[0x49cddc]
[avatargpue:1743864] [10] ../build/src/test_hydro2d_kh[0x4a495c]
[avatargpue:1743864] [11] /lib64/libstdc++.so.6(+0xc2ba3)[0x7f34a99c2ba3]
[avatargpue:1743864] [12] /lib64/libpthread.so.0(+0x814a)[0x7f34a914e14a]
[avatargpue:1743864] [13] /lib64/libc.so.6(clone+0x43)[0x7f34a8e7ddc3]
[avatargpue:1743864] *** End of error message ***

Coarse STEP 98 at t = 0.01753443829 starts ...
terminate called after throwing an instance of 'std::runtime_error'
  what():  Couldn't open file: plt00096/Level_0/Cell_D_00009
[avatargpue:1743855] *** Process received signal ***
[avatargpue:1743855] Signal: Aborted (6)
[avatargpue:1743855] Signal code:  (-6)
[avatargpue:1743855] [ 0] /lib64/libpthread.so.0(+0x12b20)[0x7f7ca35c7b20]
[avatargpue:1743855] [ 1] /lib64/libc.so.6(gsignal+0x10f)[0x7f7ca322737f]
[avatargpue:1743855] [ 2] /lib64/libc.so.6(abort+0x127)[0x7f7ca3211db5]
[avatargpue:1743855] [ 3] /lib64/libstdc++.so.6(+0x9009b)[0x7f7ca3dff09b]
[avatargpue:1743855] [ 4] /lib64/libstdc++.so.6(+0x9653c)[0x7f7ca3e0553c]
[avatargpue:1743855] [ 5] /lib64/libstdc++.so.6(+0x96597)[0x7f7ca3e05597]
[avatargpue:1743855] [ 6] /lib64/libstdc++.so.6(+0x967f8)[0x7f7ca3e057f8]
[avatargpue:1743855] [ 7] ../build/src/test_hydro2d_kh[0x435278]
[avatargpue:1743855] [ 8] ../build/src/test_hydro2d_kh[0x4697ee]
[avatargpue:1743855] [ 9] ../build/src/test_hydro2d_kh[0x49cddc]
[avatargpue:1743855] [10] ../build/src/test_hydro2d_kh[0x4a495c]
[avatargpue:1743855] [11] /lib64/libstdc++.so.6(+0xc2ba3)[0x7f7ca3e31ba3]
[avatargpue:1743855] [12] /lib64/libpthread.so.0(+0x814a)[0x7f7ca35bd14a]
[avatargpue:1743855] [13] /lib64/libc.so.6(clone+0x43)[0x7f7ca32ecdc3]
[avatargpue:1743855] *** End of error message ***
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
terminate called after throwing an instance of 'std::runtime_error'
  what():  Couldn't open file: plt00097/Level_0/Cell_D_00014
[avatargpue:1743869] *** Process received signal ***
[avatargpue:1743869] Signal: Aborted (6)
[avatargpue:1743869] Signal code:  (-6)
[avatargpue:1743869] [ 0] /lib64/libpthread.so.0(+0x12b20)[0x7ff2eaee6b20]
[avatargpue:1743869] [ 1] /lib64/libc.so.6(gsignal+0x10f)[0x7ff2eab4637f]
[avatargpue:1743869] [ 2] /lib64/libc.so.6(abort+0x127)[0x7ff2eab30db5]
[avatargpue:1743869] [ 3] /lib64/libstdc++.so.6(+0x9009b)[0x7ff2eb71e09b]
[avatargpue:1743869] [ 4] /lib64/libstdc++.so.6(+0x9653c)[0x7ff2eb72453c]
[avatargpue:1743869] [ 5] /lib64/libstdc++.so.6(+0x96597)[0x7ff2eb724597]
[avatargpue:1743869] [ 6] /lib64/libstdc++.so.6(+0x967f8)[0x7ff2eb7247f8]
[avatargpue:1743869] [ 7] ../build/src/test_hydro2d_kh[0x435278]
[avatargpue:1743869] [ 8] ../build/src/test_hydro2d_kh[0x4697ee]
[avatargpue:1743869] [ 9] ../build/src/test_hydro2d_kh[0x49cddc]
[avatargpue:1743869] [10] ../build/src/test_hydro2d_kh[0x4a495c]
[avatargpue:1743869] [11] /lib64/libstdc++.so.6(+0xc2ba3)[0x7ff2eb750ba3]
[avatargpue:1743869] [12] /lib64/libpthread.so.0(+0x814a)[0x7ff2eaedc14a]
[avatargpue:1743869] [13] /lib64/libc.so.6(clone+0x43)[0x7ff2eac0bdc3]
[avatargpue:1743869] *** End of error message ***
--------------------------------------------------------------------------
mpirun noticed that process rank 9 with PID 0 on node avatargpue exited on signal 6 (Aborted).
--------------------------------------------------------------------------

Are these errors possibly related in any way?

[BenWibking]

Ah, never mind. This error message looks like a disk quota issue.