get_extrn_ics task spuriously succeeds without getting external model files

[gsketefian]

Expected behavior

The get_extrn_ics task should report back a failure if it isn't able to retrieve a specified set of files (e.g. grib2 files) from an archive (e.g. tar) file.

Current behavior

The get_extrn_ics task runs an htar -xvf ... command, e.g.:

htar -xvf /NCEPPROD/hpssprod/runhistory/rh2019/201905/20190520/gpfs_hps_nco_ops_com_gfs_prod_gfs.2019052000.pgrb2_0p25.tar ./gfs.20190520/00/atmos/gfs.t00z.pgrb2.0p25.f000

This doesn't return the requested file because the path within the tar file (specified by the 2nd argument to htar, in this case ./gfs.20190520/00/atmos/gfs.t00z.pgrb2.0p25.f000) is incorrect (it should just be ./gfs.t00z.pgrb2.0p25.f000, i.e. without the leading directories). Nevertheless, htar returns a success:

[connecting to hpsscore1.fairmont.rdhpcs.noaa.gov/1217]
HTAR: HTAR SUCCESSFUL

(Note that if it had found the file, it would be listed in the message below.). Thus, the ex-script exregional_get_extrn_mdl_files.sh thinks the task succeeded even though it wasn't able to get the grib2 file(s), and rocoto thinks it succeeded. Then when the make_ics_mem000 task comes around, it fails because it can't find the grib2 file, and the user has to look carefully at the log file from make_ics_mem000 to find that the error is a missing file and trace it back to the get_extrn_ics task.

Machines affected

Definitely Hera, but probably all.

Steps To Reproduce

Here is the path on Hera to an example run that encounters this problem:

/scratch2/BMC/det/Gerard.Ketefian/UFS_CAM/tmp2/expt_dirs/test_02/MET_verification_only_vx_Robby.get_extrn_ics_falsely_succeeds

Detailed Description of Fix (optional)

I remember having to deal with this issue in the old shell version of the workflow. but I only vaguely remember what I did to get around it. I think it was something like getting the whole contents of the archive (e.g. using htar -tvf ...) and making sure all the files we want to retrieve (including their paths relative to the archive) exist in that list.

[mkavulich]

This appears to be the same issue as #596, just with the get_ext_ics task rather than the get_ext_lbcs task. Hopefully a solution will resolve both of those issues.

[MichaelLueken]

Issue #1165 supersedes this issue. Closing this issue now.