FatesColdAllVars testmod failing due to invalid memory reference during history write

[glemieux]

Running SMS_D_Lm3.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdAllVars is resulting in a seg fault during history write:

lnd.log:

5046  clm: leaving fates model           1           3
5047 
5048  hist_htapes_wrapup : Writing current time sample to local history file
5049  ./SMS_D_Lm3.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdAllV
5050  ars.0801-160438de.clm2.h0.2000-01-01-00000.nc at nstep =         3168
5051   for history time interval beginning at    65.0000000000000       and ending at
5052      66.0000000000000

cesm.log:

  3 dec0453.hsn.de.hpc.ucar.edu 129: Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
  4 dec0453.hsn.de.hpc.ucar.edu 129:
  5 dec0453.hsn.de.hpc.ucar.edu 129: Backtrace for this error:
  6 dec0453.hsn.de.hpc.ucar.edu 129: #0  0x1482cd5ead4f in ???
  7 dec0453.hsn.de.hpc.ucar.edu 129:        at /usr/src/debug/glibc-2.31-150300.41.1.x86_64/signal/../sysdeps/unix/sysv/linux/x86_64/sigaction.c:0
  8 dec0453.hsn.de.hpc.ucar.edu 129: #1  0x1482cd720d8e in ???
  9 dec0453.hsn.de.hpc.ucar.edu 129:        at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:325
 10 dec0453.hsn.de.hpc.ucar.edu 129: #2  0x1482daccf6a9 in PIOc_write_darray_multi
 11 dec0453.hsn.de.hpc.ucar.edu 129:        at /glade/derecho/scratch/jedwards/tmp/spack-stage/spack-stage-parallelio-2.6.2-nvj5ecxzg4ol3brhepxstfccdoi57fat/spack-src/src/clib/pio_darray.c:381
 12 dec0453.hsn.de.hpc.ucar.edu 129: #3  0x1482dacd6316 in flush_buffer
 13 dec0453.hsn.de.hpc.ucar.edu 129:        at /glade/derecho/scratch/jedwards/tmp/spack-stage/spack-stage-parallelio-2.6.2-nvj5ecxzg4ol3brhepxstfccdoi57fat/spack-src/src/clib/pio_darray_int.c:1820
 14 dec0453.hsn.de.hpc.ucar.edu 129: #4  0x1482dacd1243 in PIOc_write_darray
 15 dec0453.hsn.de.hpc.ucar.edu 129:        at /glade/derecho/scratch/jedwards/tmp/spack-stage/spack-stage-parallelio-2.6.2-nvj5ecxzg4ol3brhepxstfccdoi57fat/spack-src/src/clib/pio_darray.c:777
 16 dec0453.hsn.de.hpc.ucar.edu 129: #5  0x1482daf55981 in write_darray_1d_cinterface_double
 17 dec0453.hsn.de.hpc.ucar.edu 129:        at /glade/derecho/scratch/jedwards/tmp/spack-stage/spack-stage-parallelio-2.6.2-nvj5ecxzg4ol3brhepxstfccdoi57fat/spack-src/src/flib/piodarray.F90.in:134
 18 dec0453.hsn.de.hpc.ucar.edu 129: #6  0x1482daf5416a in __piodarray_MOD_write_darray_1d_double
 19 dec0453.hsn.de.hpc.ucar.edu 129:        at /glade/derecho/scratch/jedwards/tmp/spack-stage/spack-stage-parallelio-2.6.2-nvj5ecxzg4ol3brhepxstfccdoi57fat/spack-src/src/flib/piodarray.F90.in:241
 20 dec0453.hsn.de.hpc.ucar.edu 129: #7  0x1482daf4f312 in __piodarray_MOD_write_darray_2d_double
 21 dec0453.hsn.de.hpc.ucar.edu 129:        at /glade/derecho/scratch/jedwards/tmp/spack-stage/spack-stage-parallelio-2.6.2-nvj5ecxzg4ol3brhepxstfccdoi57fat/spack-src/src/flib/piodarray.F90.in:293
 22 dec0453.hsn.de.hpc.ucar.edu 129: #8  0x5b830c in __ncdio_pio_MOD_ncd_io_2d_double
 23 dec0453.hsn.de.hpc.ucar.edu 129:        at /glade/u/home/glemieux/ctsm/src/main/ncdio_pio.F90.in:2033
 24 dec0453.hsn.de.hpc.ucar.edu 129: #9  0x6a0754 in hfields_write
 25 dec0453.hsn.de.hpc.ucar.edu 129:        at /glade/u/home/glemieux/ctsm/src/main/histFileMod.F90:3704
 26 dec0453.hsn.de.hpc.ucar.edu 129: #10  0x6c4886 in __histfilemod_MOD_hist_htapes_wrapup
 27 dec0453.hsn.de.hpc.ucar.edu 129:        at /glade/u/home/glemieux/ctsm/src/main/histFileMod.F90:4225
 28 dec0453.hsn.de.hpc.ucar.edu 129: #11  0x5da3dd in __clm_driver_MOD_clm_drv
 29 dec0453.hsn.de.hpc.ucar.edu 129:        at /glade/u/home/glemieux/ctsm/src/main/clm_driver.F90:1448
 30 dec0453.hsn.de.hpc.ucar.edu 129: #12  0x58bba7 in modeladvance
 31 dec0453.hsn.de.hpc.ucar.edu 129:        at /glade/u/home/glemieux/ctsm/src/cpl/nuopc/lnd_comp_nuopc.F90:904
 32 dec0453.hsn.de.hpc.ucar.edu 129: #13  0x1482d7feb99f in _ZNK5ESMCI13MethodElement7executeEPvPi
 33 dec0453.hsn.de.hpc.ucar.edu 129:        at /glade/derecho/scratch/jedwards/tmp/spack-stage/spack-stage-esmf-8.6.0-bsogfa4e7dreitxbwm4gbppisw5q4x2t/spack-src/src/Superstructure/Component/src/ESMCI_Meth    odTable.C:377
...