Intermittent failure of MOSART and elm.cbudget tests on mappy

[peterdschwartz]

ERS.r05_r05.IELM.mappy_gnu.elm-V2_ELM_MOSART_features will alternate between FAIL and DIFF with the below error message.

profile sums:    1.0000000000000000        1.0000000000000000        1.0000000000000000       0.99923994737367017     
 ENDRUN: ERROR: sum-1 > deltaERROR in /home/e3sm-jenkins/jenkins-ws/workspace/mappy_e3sm_master/E3SM/components/elm/src/biogeochem/VerticalProfileMod.F90 at line 340                                                                                                                                                                                                                                                                                                                                                                                        
 ERROR: Unknown error submitted to shr_abort_abort

[peterdschwartz]

ERS.r05_r05.ICNPRDCTCBC.mappy_gnu.elm-cbudget fails in the same way

Edit:
Error message from 09/24/2024

 profile sums:   0.89219646033256605        1.0000000000000002        1.0000000000000000        1.0000000000000000     
 ENDRUN: ERROR: sum-1 > deltaERROR in /home/e3sm-jenkins/jenkins-ws/workspace/mappy_e3sm_next/E3SM/components/elm/src/biogeochem/VerticalProfileMod.F90 at line 340                                                                                                                                                                                                                                                                                                                                                                                          
 ERROR: Unknown error submitted to shr_abort_abort.

[peterdschwartz]

Note for 10/03/2024
ERS.r05_r05.ICNPRDCTCBC.mappy_gnu.elm-cbudget log shows a segmentation fault message instead of the error with profiles in VeritcalProfileMod.

HOWEVER, on mappy master SMS.r05_r05.I1850ELMCN.mappy_gnu.elm-qian_1948
FAILED with

 profile sums:    1.0000000000000000        1.0000000000000000        1.0000000000000000       0.95524140840331506     
 ENDRUN: ERROR: sum-1 > deltaERROR in /home/e3sm-jenkins/jenkins-ws/workspace/mappy_e3sm_master/E3SM/components/elm/src/biogeochem/VerticalProfileMod.F90 at line 340                                                                                                                                                                                                                                                                                                                                                                                        
 ERROR: Unknown error submitted to shr_abort_abort.

[peterdschwartz]

more data from elm-cbudget due to that PR.
will look into it later but quick note is that wtcol(p) = 0.0 so this should be the call of decomp_vertprofiles in elm_drv instead of in EcosystemDynMod

 profile sums:    1.0000000000000000        1.5100795568461205        1.0000000000000000        1.0000000000000000     
 c:       702769
 altmax_lastyear_indx:           15
 cinput_rootfr:    3.6474236958527042        3.3458810358498630        2.9114856288385891        2.3191019760804545        1.6011705081532086       0.87937331399225993       0.33654208396518998        7.3628037262981408E-002   6.9445051230372055E-003   1.9679511245244402E-004   0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000     
 dzsoi_decomp:    1.7512817916255204E-002   2.7578969259676251E-002   4.5470033242413201E-002   7.4967410986208557E-002  0.12360036510228053       0.20378255101043175       0.33598062644843263       0.55393840536868488       0.91329003158906108        1.5057607013992766        2.4825796969813321        4.0930819526214002        6.7483512780057175        11.126150294204420        13.851152141963599     
 surface_prof:    53.187098611515587        27.424911410440870        11.800194810449913        4.0635261684082673       0.96925050730793827       0.12619130990079749        6.0549255405471835E-003   5.6033058000951139E-005   3.4384797389171701E-008   2.4066526910736183E-013
 p, itype(p), wtcol(p):      1420563           0   0.0000000000000000     
 cinput_rootfr(p,:):    57.101033356363004        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000     
 croot_prof(p,:):    57.101033356363004        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        1.5181814565858882        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000     
 froot_prof(p,:):    57.101033356363004        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000     
 leaf_prof(p,:):    19.871329356601105        10.246271397373919        4.4086924023288736        1.5181814565858882       0.36212345780408262        4.7146566487774995E-002   2.2621918244635853E-003   2.0934613458184733E-005   1.2846567149129900E-008   8.9915392115259993E-014   0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000     
 stem_prof(p,:):    19.871329356601105        10.246271397373919        4.4086924023288736        1.5181814565858882       0.36212345780408262        4.7146566487774995E-002   2.2621918244635853E-003   2.0934613458184733E-005   1.2846567149129900E-008   8.9915392115259993E-014   0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000     
 ENDRUN: ERROR: sum-1 > deltaERROR in /home/e3sm-jenkins/jenkins-ws/workspace/mappy_e3sm_next/E3SM/components/elm/src/biogeochem/VerticalProfileMod.F90 at line 353                                                                                                                                                                                                                                                                                                                                                                                          
 ERROR: Unknown error submitted to shr_abort_abort.