Issue with Frontera spack-stack 1.6.0

[uturuncoglu]

Describe the bug
Under UFS Coastal we are supporting Frontera for the ocean modeling community. We installed 1.6.0 previously without any issue and it was working fine until recent sync with the UFS WM. It seems that fms, g2 and g2tmpl modules are upgraded but still under 1.6.0 and then this requires updating our spack stack installation with those modules.

here are the new module version found in ufs.common
fms/2024.01
g2/3.5.1
g2tmpl/1.13.0

and here are the module that we are using currently,
fms/2023.04.lua
g2/3.4.5.lua
g2tmpl/1.10.2.lua

So, my question is that how this is happening. I was expecting to have release like 1.6.1 if there is some version change in modules. At this point, it is hard for me to update my 1.6.0 branch (has front support) to install these new versions since those new versions are available in newer version of spack and that is not compatible with our spack-stack 1.6.0 branch. Anyway, please let me know if there is 1.6.0 branch in Spack-stack side that has all those updated modules. If not, I think the kind of updates will create more headache in the UFS Coastal side in the future and somehow we need to find a consistent way to handle those kind of issues.

To Reproduce
Steps to reproduce the behavior:

• clone UFS Coastal on Frontera
  git clone --recursive https://github.com/oceanmodeling/ufs-weather-model.git
• try to build one of the configuration
  cd ufs-weather-model/tests
  ./compile.sh frontera "-DAPP=CSTLS -DUSE_ATMOS=ON -DNO_PARAMETIS=OFF -DOLDIO=ON" coastal intel NO NO

Expected behavior
A clear and concise description of what you expected to happen.
expecting to build the configuration

System:
What system(s) are you running the code on?
on Frontera

Additional context
Add any other context about the problem here.
No

[climbfuji]

@uturuncoglu Because some groups are less agile than others (NOAA/UFS, do you hear me?), they do not want to update spack-stack whenever a new release is made. But at the same time, they request updates to specific packages being made available in the existing stack. Bad practice imo, but that's what they insist on. Therefore, when there are no conflicts between packages and modules (especially duplicate module names), we go back and install new versions IN the existing environment OR as an addon/chained environment. The latter is preferred, since it leaves the original environment as-is.

The process generally is like this:

1. Update the local clone of your spack-stack-1.6.0 install from the release/1.6.0 branches upstream (spack submodule and spack-stack parent).
2. Create a chained environment that reuses whatever possible from the original environment and only installs the new packages.
3. Run the install as usual, and add --upstream-modules to the spack module [tcl|lmod] refresh command.

I know this is unsatisfying, and I agree with you that this isn't a good situation. But NOAA&UFS want it this way.

[uturuncoglu]

@climbfuji Thanks for your help. Since UFS WM is used among different applications, it would be nice to follow some kind of standards in dependencies. I am not talking how community will adjust their self to these kind of changes. It is way more complicated for them. At this point, I solved the issue in UFS Coastal side by modifying Frontra module file and not pointing ufs.common in there (I know it is ugly solution but the fatest one). So, I could use the old version of those packages. In the meantime, I could try to follow the same approach like chaining and see what happens. Do you have detailed instruction of it? BTW, the spack that is found in spack-stack 1.6.0 branch is old and does not have newer version of those packages. When I try to update spack submodule, then it was giving error for me and complaining due to I think the inconsistencies between spack-stack and used version of spack. So, it would be time consuming to try installing newer version of those packages using spack-stack 1.6.0 as a base.

[climbfuji]

The "instructions" are here. It's a slightly different approach and requires copying the environment definition from Orion to Frontera: #1180 (comment)