Gaea C6 support for UFSWM #2448

BrianCurtis-NOAA · 2024-10-02T17:48:48Z

Commit Queue Requirements:

Fill out all sections of this template.
All sub component pull requests have been reviewed by their code managers.
Run the full Intel+GNU RT suite (compared to current baselines) on either Hera/Derecho/Hercules
Commit 'test_changes.list' from previous step

Description:

This PR will bring in all changes necessary to provide Gaea C6 support for UFSWM

Commit Message:

* UFSWM - Gaea C6 Support

Priority:

Normal

Git Tracking

UFSWM:

Closes #
None

Sub component Pull Requests:

None

UFSWM Blocking Dependencies:

Blocked by #
None

Changes

Regression Test Changes (Please commit test_changes.list):

No Baseline Changes. (just adds logs for Gaea C6)

Input data Changes:

None.

Library Changes/Upgrades:

No Updates

Testing Log:

BrianCurtis-NOAA · 2024-10-02T17:53:59Z

cpld_control_p8 intel fails for timing out, so there's work to tweak the configs to better match the C6 hardware.

I think there's still lots of other items to check here, this is just a placeholder for now. Please feel free to send PR's to my fork/branch to add/adjust/fix any issues etc...

…into gaeac6

BrianCurtis-NOAA · 2024-10-02T17:55:49Z

Also, once things start falling into place, we'll need to make sure intelllvm support is available for c6.

RatkoVasic-NOAA · 2024-10-04T00:50:55Z

@BrianCurtis-NOAA, name change suggestion:

gaea -----> gaea-c5
gaeac6 ---> gaea-c6

sanAkel · 2024-10-04T01:01:09Z

@BrianCurtis-NOAA Shall I re-try building with these modulefiles/ufs_gaeac6.intel.lua in this PR?

DusanJovic-NOAA · 2024-10-04T13:43:02Z

tests/compile.sh

@@ -95,7 +98,7 @@ export SUITES
 set -ex

 # Valid applications
-if [[ ${MACHINE_ID} != gaea ]] || [[ ${RT_COMPILER} != intelllvm ]]; then # skip MOM6SOLO on gaea with intelllvm
+if [[ ${MACHINE_ID} != gaea-c5 && ${MACHINE_ID} != gaea-c6 ]] || [[ ${RT_COMPILER} != intelllvm ]]; then # skip MOM6SOLO on gaea with intelllvm


Why do we even need this logic here, adding or not adding -DMOM6SOLO=ON? As far as I know, we do not regression test MOM6SOLO. Can we remove this block of code entirely from this script?

Good question. It was added there for a reason, and I don't recall if we ever RT'd MOM6SOLO. @junwang-noaa do you recall what this block of code was used for?

If I remember correctly, this is to support standalone MOM testing. @jiandewang Do you know why MOM6 SOLO does not work on gaea?

Hi! I'm new to the UFS, but AFAIK, nobody seems to use -DMOM6SOLO=ON, though I would differ it to @junwang-noaa.

@junwang-noaa My understanding from @jiandewang is that he (and others) are no longer routinely testing MOM solo config; I have always built using instructions at MOM6-examples

If I remember correctly, this is to support standalone MOM testing. @jiandewang Do you know why MOM6 SOLO does not work on gaea?

It was added here many years ago and we never tried this SOLO on any platform. My understanding is with nuopc_cap it has to be coupled with something.

@junwang-noaa My understanding from @jiandewang is that he (and others) are no longer routinely testing MOM solo config; I have always built using instructions at MOM6-examples

yes we use MOM-example to do standalone test when it's needed to do some debug work (to help GFDL to narrow down issue when their big PR is not working as expected in UWM).

So you do not use tests/compile.sh to build standalone test, is that correct?

So you do not use tests/compile.sh to build standalone test, is that correct?

correct

Then we should remove it from compile.sh

BrianCurtis-NOAA · 2024-10-04T14:12:49Z

cpld_control_p8 fails with:

  5: MPICH ERROR [Rank 5] [job id 207188364.0] [Fri Oct  4 13:33:08 2024] [c6n0210] - Abort(941244175) (rank 5 in comm 0): Fatal error in PMPI_Win_create: Other MPI error, error stack:
  5: PMPI_Win_create(294)......: MPI_Win_create(base=0x7ffe81f20fe0, size=0, disp_unit=1, MPI_INFO_NULL, comm=0xc4000060, win=0x7ffe81f2113c) failed
  5: MPID_Win_create(89).......:
  5: MPIDIG_mpi_win_create(872):
  5: win_allgather(246)........: OFI mr_enable failed (ofi_win.c:246:win_allgather:Address already in use)

and control_p8 runs to completion:

0: * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . 
  0: *****************RESOURCE STATISTICS*******************************
  0: The total amount of wall time                        = 853.216145
  0: The total amount of time in user mode                = 216.242551
  0: The total amount of time in sys mode                 = 410.041583
  0: The maximum resident set size (KB)                   = 1720560
  0: Number of page faults without I/O activity           = 131391
  0: Number of page faults with I/O activity              = 173
  0: Number of times filesystem performed INPUT           = 1024
  0: Number of times filesystem performed OUTPUT          = 0
  0: Number of Voluntary Context Switches                 = 16903
  0: Number of InVoluntary Context Switches               = 9006
  0: *****************END OF RESOURCE STATISTICS*************************

BrianCurtis-NOAA · 2024-10-04T14:51:32Z

@DusanJovic-NOAA this look ok?:

diff --git a/tests/compile.sh b/tests/compile.sh
index 2c3c7796..26e3a788 100755
--- a/tests/compile.sh
+++ b/tests/compile.sh
@@ -97,17 +97,6 @@ SUITES=$(grep -Po "\-DCCPP_SUITES=\K[^ ]*" <<< "${MAKE_OPT}")
 export SUITES
 set -ex
 
-# Valid applications
-if [[ ${MACHINE_ID} != gaea-c5 && ${MACHINE_ID} != gaea-c6 ]] || [[ ${RT_COMPILER} != intelllvm ]]; then # skip MOM6SOLO on gaea with intelllvm
-  if [[ "${MAKE_OPT}" == *"-DAPP=S2S"* ]]; then
-      CMAKE_FLAGS+=" -DMOM6SOLO=ON"
-  fi
-
-  if [[ "${MAKE_OPT}" == *"-DAPP=NG-GODAS"* ]]; then
-      CMAKE_FLAGS+=" -DMOM6SOLO=ON"
-  fi
-fi
-
 CMAKE_FLAGS=$(set -e; trim "${CMAKE_FLAGS}")
 echo "CMAKE_FLAGS = ${CMAKE_FLAGS}"

DusanJovic-NOAA · 2024-10-04T15:03:16Z

@DusanJovic-NOAA this look ok?:

diff --git a/tests/compile.sh b/tests/compile.sh
index 2c3c7796..26e3a788 100755
--- a/tests/compile.sh
+++ b/tests/compile.sh
@@ -97,17 +97,6 @@ SUITES=$(grep -Po "\-DCCPP_SUITES=\K[^ ]*" <<< "${MAKE_OPT}")
 export SUITES
 set -ex
 
-# Valid applications
-if [[ ${MACHINE_ID} != gaea-c5 && ${MACHINE_ID} != gaea-c6 ]] || [[ ${RT_COMPILER} != intelllvm ]]; then # skip MOM6SOLO on gaea with intelllvm
-  if [[ "${MAKE_OPT}" == *"-DAPP=S2S"* ]]; then
-      CMAKE_FLAGS+=" -DMOM6SOLO=ON"
-  fi
-
-  if [[ "${MAKE_OPT}" == *"-DAPP=NG-GODAS"* ]]; then
-      CMAKE_FLAGS+=" -DMOM6SOLO=ON"
-  fi
-fi
-
 CMAKE_FLAGS=$(set -e; trim "${CMAKE_FLAGS}")
 echo "CMAKE_FLAGS = ${CMAKE_FLAGS}"

Yes.

initial testing to get UFSWM working on Gaea C6

b968b96

Merge branch 'develop' of github.com:ufs-community/ufs-weather-model …

efe342e

…into gaeac6

jkbk2004 mentioned this pull request Oct 4, 2024

Enable ufs-weather-model on Gaea-C6 #2407

Open

BrianCurtis-NOAA added 4 commits October 4, 2024 08:53

gaea->gaea-c5 and gaeac6->gaea-c6

7476837

Fixed linter issue

742a7c2

Update to 192 cores on Gaea-c6

5bee5b2

Update tests to gaea-c5 and added gaea-c6 where necessary

63a56ac

DusanJovic-NOAA reviewed Oct 4, 2024

View reviewed changes

Remove MOM6SOLO from compile.sh

bb83396

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Gaea C6 support for UFSWM #2448

Gaea C6 support for UFSWM #2448

BrianCurtis-NOAA commented Oct 2, 2024

BrianCurtis-NOAA commented Oct 2, 2024

BrianCurtis-NOAA commented Oct 2, 2024

RatkoVasic-NOAA commented Oct 4, 2024

sanAkel commented Oct 4, 2024

DusanJovic-NOAA Oct 4, 2024

BrianCurtis-NOAA Oct 4, 2024

junwang-noaa Oct 4, 2024

sanAkel Oct 4, 2024

sanAkel Oct 4, 2024 •

edited

Loading

jiandewang Oct 4, 2024

jiandewang Oct 4, 2024

DusanJovic-NOAA Oct 4, 2024

jiandewang Oct 4, 2024

DusanJovic-NOAA Oct 4, 2024 •

edited

Loading

BrianCurtis-NOAA commented Oct 4, 2024

BrianCurtis-NOAA commented Oct 4, 2024

DusanJovic-NOAA commented Oct 4, 2024

Gaea C6 support for UFSWM #2448

Are you sure you want to change the base?

Gaea C6 support for UFSWM #2448

Conversation

BrianCurtis-NOAA commented Oct 2, 2024

Commit Queue Requirements:

Description:

Commit Message:

Priority:

Git Tracking

UFSWM:

Sub component Pull Requests:

UFSWM Blocking Dependencies:

Changes

Regression Test Changes (Please commit test_changes.list):

Input data Changes:

Library Changes/Upgrades:

Testing Log:

BrianCurtis-NOAA commented Oct 2, 2024

BrianCurtis-NOAA commented Oct 2, 2024

RatkoVasic-NOAA commented Oct 4, 2024

sanAkel commented Oct 4, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sanAkel Oct 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DusanJovic-NOAA Oct 4, 2024 • edited Loading

Choose a reason for hiding this comment

BrianCurtis-NOAA commented Oct 4, 2024

BrianCurtis-NOAA commented Oct 4, 2024

DusanJovic-NOAA commented Oct 4, 2024

sanAkel Oct 4, 2024 •

edited

Loading

DusanJovic-NOAA Oct 4, 2024 •

edited

Loading