Add scotch library to hpc-stack #504

ulmononian · 2022-11-09T19:03:52Z

A request to support the scotch library (on at least Hera and Orion) came from @aliabdolali via #501. This PR includes the pertinent additions/adaptations to enable installation of scotch as part of the stack, including:

a new build_scotch.sh script in /libs
the addition of "build_lib scotch" to the ufs 3rd party apps. section of build_stack.sh
a scotch lua modulefile template

Successful installation as a part of a test hpc-stack was performed on Hera w/ Intel-2022.1.2 compiler and MPI (src: /scratch1/NCEPDEV/stmp2/Cameron.Book/hpcs_work/src/scotchNsplenda , lib: /scratch1/NCEPDEV/stmp2/Cameron.Book/hpcs_work/libs/scotch). @natalie-perlin will follow with Orion testing, as I do not have access.

I have not added a scotch section to any of the stack/*yaml files simply because I am not sure which would be most relevant. Open to suggestions, or we can just leave the inclusion of the scotch section to users who actually want/need the library in their stack.

Perhaps @aliabdolali may want to test loading the scotch-7.0.1 module to see if it works as expected in context (e.g., within WW3, etc.).

If the basic additions in this PR are deemed OK, we can go ahead and install scotch in the forthcoming, official, EPIC-maintained hpc-stack locations on Hera and Orion.

Note: I would like to recognize contributions from @natalie-perlin to this work.

…ersion_id envar, etc..

…version was a mistake.

Remove loading of netcdf and cmake in module section. Not needed.

natalie-perlin · 2022-11-10T01:28:11Z

libs/build_scotch.sh

+mkdir -p build && cd build
+
+# Compile & Install Scotch/PTscotch
+cmake VERBOSE=1 -DCMAKE_Fortran_COMPILER=ifort -DCMAKE_C_COMPILER=icc -DCMAKE_INSTALL_PREFIX=$prefix -DCMAKE_BUILD_TYPE=Release ..


We need to avoid hard-coding compiler names (such as ifort and icc), because it needs to general for different compilers.

natalie-perlin · 2022-11-10T01:40:55Z

A similar scotch.lua template modulefile is needed for the case of serial installation, in ./compiler/compilerName/compilerVersion/scotch/ . It is very similar to the one for the mpi,
except for the part:

local hierA = hierarchyA(pkgNameVer,1)
local compNameVer = hierA[1]
local compNameVerD = compNameVer:gsub("/","-")

…/compilerVersion for cases of serial installation.

ulmononian · 2022-11-10T15:48:43Z

A similar scotch.lua template modulefile is needed for the case of serial installation, in ./compiler/compilerName/compilerVersion/scotch/ . It is very similar to the one for the mpi, except for the part:

local hierA = hierarchyA(pkgNameVer,1) local compNameVer = hierA[1] local compNameVerD = compNameVer:gsub("/","-")

done. please see: git commit -m "Add scotch modulefile ito hpc-stack/modulefiles/compiler/compilerName/compilerVersion for cases of serial installation.".

edwardhartnett · 2022-11-10T15:49:47Z

We should not be adding things to hpc-stack, this project is due to be retired very soon.

All new work should be happening is the spack-stack project.

ulmononian · 2022-11-10T15:52:47Z

We should not be adding things to hpc-stack, this project is due to be retired very soon.

All new work should be happening is the spack-stack project.

I understand this concern, @edwardhartnett . However, how are we to address requests for hpc-stack additions while developers are still utilizing this stack and spack-stack is not officially adopted or in use on the UFS-WM side? The transition process seems slightly nebulous.

edwardhartnett · 2022-11-10T15:54:16Z

Is this also being added to spack-stack?

ulmononian · 2022-11-10T15:56:08Z

Is this also being added to spack-stack?

It certainly can be (and the latest scotch is available through spack, as I tested an install of it: /scratch1/NCEPDEV/stmp2/Cameron.Book/spack_work/spack-stack/envs/scotch.intel), but a request to add it to a specific spack-stack environment/configuration has not been submitted.

aliabdolali · 2022-11-10T16:41:41Z

Hi @ulmononian

I tried WW3 with the scotch you compiled on Hera using the following modules:

module purge
module load cmake/3.20.1
module use module use /scratch1/NCEPDEV/stmp2/Cameron.Book/hpcs_work/libs/scotch/modulefiles/stack/
module load hpc/1.2.0
module load hpc-intel/2022.1.2
module load hpc-impi/2022.1.2
module load scotch
module load netcdf/4.9.0
module load jasper/2.0.25
module load zlib/1.2.12
module load libpng/1.6.37
module load hdf5/1.12.2
module load g2lib/1.4.0

We also need the following libraries, but I could not load them due to the conflict with the previously loaded libs. Could you guide how we should proceed?

module load bacio
module load w3emc
module load esmf

We also need to define SCOTCH_ROOT, similar to NETCDF_ROOT. Can we do it?

export SCOTCH_PATH=/scratch1/NCEPDEV/stmp2/Cameron.Book/hpcs_work/libs/scotch/intel-2022.1.2/impi-2022.1.2/scotch/7.0.1/

With the loaded libs and in absence of bacio, w3emc and esmf, I could run one test successfully that does not require those dependencies.

climbfuji · 2022-11-10T16:48:20Z

Is this also being added to spack-stack?

The issue is JCSDA/spack-stack#336 - nobody has worked on it yet.

ulmononian · 2022-11-10T16:53:23Z

Hi @ulmononian

I tried WW3 with the scotch you compiled on Hera using the following modules:
module purge
module load cmake/3.20.1
module use module use /scratch1/NCEPDEV/stmp2/Cameron.Book/hpcs_work/libs/scotch/modulefiles/stack/
module load hpc/1.2.0
module load hpc-intel/2022.1.2
module load hpc-impi/2022.1.2
module load scotch
module load netcdf/4.9.0
module load jasper/2.0.25
module load zlib/1.2.12
module load libpng/1.6.37
module load hdf5/1.12.2
module load g2lib/1.4.0
We also need the following libraries, but I could not load them due to the conflict with the previously loaded libs. Could you guide how we should proceed?
module load bacio
module load w3emc
module load esmf
We also need to define SCOTCH_ROOT, similar to NETCDF_ROOT. Can we do it?

export SCOTCH_PATH=/scratch1/NCEPDEV/stmp2/Cameron.Book/hpcs_work/libs/scotch/intel-2022.1.2/impi-2022.1.2/scotch/7.0.1/

With the loaded libs and in absence of bacio, w3emc and esmf, I could run one test successfully that does not require those dependencies.

@aliabdolali thanks for taking a look at this. let me get scotch installed in a stack with all of the libraries you mentioned. my intial test stack was simply to get scotch installed w/ its own dependencies, but i did not think about what scotch would need within ww3 for testing. my apologies!

i will also look into the SPACK_ROOT/SPACK_PATH issue that you mentioned. is this based on some issue you had when loading and running ww3 w/ the scotch module>

edwardhartnett · 2022-11-10T18:11:02Z

Please do not get something working in hpc-stack which is not also working in spack-stack.

Otherwise, when it is time to switch from hpc-stack to spack-stack, we will not be able to do so.

ulmononian · 2022-11-10T18:15:54Z

Please do not get something working in hpc-stack which is not also working in spack-stack.

Otherwise, when it is time to switch from hpc-stack to spack-stack, we will not be able to do so.

Fair enough, but how is this to be applied in practice when there is a request and need for software before the spack-stack is adopted by the UFS community? for example, what would the appropriate avenue for addressing #501, wherein ww3 development (and gfs downstream) requires scotch in an official stack?

edwardhartnett · 2022-11-10T18:18:31Z

Get it working in both hpc-stack and spack-stack. Getting it working only in hpc-stack is not good. We will not be able to retire hpc-stack if it contains software not available in spack-stack.

ulmononian · 2022-11-10T18:40:55Z

@aliabdolali i just added bacio, w3emc, and esmf (and their required dependencies) to /scratch1/NCEPDEV/stmp2/Cameron.Book/hpcs_work/libs/scotch/modulefiles/stack. i was able to load everything you mentioned w/:

module purge
module load cmake/3.20.1
module use module use /scratch1/NCEPDEV/stmp2/Cameron.Book/hpcs_work/libs/scotch/modulefiles/stack/
module load hpc/1.2.0
module load hpc-intel/2022.1.2
module load hpc-impi/2022.1.2
module load scotch
module load netcdf/4.9.0
module load jasper/2.0.25
module load zlib/1.2.12
module load libpng/1.6.37
module load hdf5/1.12.2
module load g2lib/1.4.0
module load bacio
module load w3emc
module load esmf

please let me know if this allows you to run scotch as required within ww3.

ulmononian · 2022-11-10T19:13:52Z

Get it working in both hpc-stack and spack-stack. Getting it working only in hpc-stack is not good. We will not be able to retire hpc-stack if it contains software not available in spack-stack.

that is understandable. i would be glad to also demonstrate that scotch works as part of of the spack-stack. as i said, i did install it already ( /scratch1/NCEPDEV/stmp2/Cameron.Book/spack_work/spack-stack/envs/scotch.intel), but if @edwardhartnett or @climbfuji have a specific installation config/testing in mind to validate scotch within spack-stack, i can try to handle it.

aliabdolali · 2022-11-10T19:17:15Z

@aliabdolali i just added bacio, w3emc, and esmf (and their required dependencies) to /scratch1/NCEPDEV/stmp2/Cameron.Book/hpcs_work/libs/scotch/modulefiles/stack. i was able to load everything you mentioned w/:
module purge
module load cmake/3.20.1
module use module use /scratch1/NCEPDEV/stmp2/Cameron.Book/hpcs_work/libs/scotch/modulefiles/stack/
module load hpc/1.2.0
module load hpc-intel/2022.1.2
module load hpc-impi/2022.1.2
module load scotch
module load netcdf/4.9.0
module load jasper/2.0.25
module load zlib/1.2.12
module load libpng/1.6.37
module load hdf5/1.12.2
module load g2lib/1.4.0
module load bacio
module load w3emc
module load esmf
please let me know if this allows you to run scotch as required within ww3.

Thanks @ulmononian
I was able to test it successfully but would like WW3 code managers (@MatthewMasarik-NOAA @JessicaMeixner-NOAA) to confirm it would satisfy the rest of our tests.
FYI: @AvichalMehra-NOAA

ulmononian · 2022-11-10T19:19:22Z

@aliabdolali as for the SCOTCH_ROOT question, i am not sure if i understand the issue. $scotch_ROOT is defined via

setenv("scotch_ROOT","/scratch1/NCEPDEV/stmp2/Cameron.Book/hpcs_work/libs/scotch/intel-2022.1.2/impi-2022.1.2/scotch/7.0.1")

while its path is set via

prepend_path("PATH","/scratch1/NCEPDEV/stmp2/Cameron.Book/hpcs_work/libs/scotch/intel-2022.1.2/impi-2022.1.2/scotch/7.0.1/bin")

i've attached a screenshot to show what the paths/envars from the modulefile are read as for the current scotch location you are testing. please let me know if there is something done w/ the netcdf lua file that you need in the scotch lua file.

ulmononian · 2022-11-10T19:24:54Z

@aliabdolali: looking forward to hearing how the tests go for @MatthewMasarik-NOAA & @JessicaMeixner-NOAA.

JessicaMeixner-NOAA · 2022-11-10T19:28:58Z

@ulmononian we're waiting for some CMAKE issues using SCOTCH within WW3 to be resolved before we test this. Just wanted to keep you in that loop. We'll test this installation of SCOTCH as soon as those issues are resolved.

ulmononian · 2022-11-10T19:53:16Z

@JessicaMeixner-NOAA thanks for letting me know!

and speaking of cmake: @aliabdolali what was the reason you ended up including -DCMAKE_Fortran_COMPILER=ifort -DCMAKE_C_COMPILER=icc in the cmake section of your build recipe? i have tested and can build on hera without hard-coding these compiler flags, but thought maybe you had a motivation for this.

aliabdolali · 2022-11-10T19:55:07Z

@JessicaMeixner-NOAA thanks for letting me know!

and speaking of cmake: @aliabdolali what was the reason you ended up including -DCMAKE_Fortran_COMPILER=ifort -DCMAKE_C_COMPILER=icc in the cmake section of your build recipe? i have tested and can build on hera without hard-coding these compiler flags, but thought maybe you had a motivation for this.

There is no reason for it, I wanted to make sure icc and ifort are being used, I think we are good to go if you were able to compile it without them.

aerorahul · 2023-04-28T17:20:08Z

Should this be closed?

natalie-perlin · 2023-05-15T21:53:50Z

A scotch library 7.0.3 was rebuild and installed as a part of the hpc-stack on Hera and Orion, with the following options:

cmake VERBOSE=1 -DCMAKE_Fortran_COMPILER=mpiifort
-DCMAKE_C_COMPILER=mpiicc -DMAKE_CXX_COMPILER=mpiicpc
-DCMAKE_INSTALL_PREFIX=/work/noaa/epic-ps/role-epic-ps/hpc-stack/libs/intel-2022.1.2/intel-2022.1.2/impi-2022.1.2/scotch/7.0.3 -DCMAKE_BUILD_TYPE=Release -DTHREADS=OFF -DMPI_THREAD_MULTIPLE=OFF ..

(#501 (comment))

It cannot be easily added as a standard part of the hpc-stack, however, in particular for the current intel compilers, because it requires an updated gcc. The gcc as a part of intel compiler suite is outdated; gcc from GNU compiler suite could be used, but it requires modifying GNU modulefile to avoid conflicts with Intel compiler.
Modified GNU compiler modulefiles were used on Orion and Hera.
For the general user and not on NOAA RDHPCs, having a specific compiler as defined in config_custom.sh would be best.

ulmononian and others added 9 commits November 7, 2022 20:31

Add initial build script for scotch library to support marine/WW3.

fe0faa5

Update scotch build script.

66e32a4

Add scotch build_lib command to build_stack.sh.

8d72c96

Update build_scotch.sh w/ proper prefix path, module load commands, v…

24155d1

…ersion_id envar, etc..

Add scotch.lua modulefile.

fc25e21

Add section to remove pre-existing build directory in build_scotch.sh.

6fb0fcc

Revert config_hera.sh to develop version. Initial commit of modified …

1a9fea0

…version was a mistake.

Change version/id variables to match convention.

cc9abda

Update build_scotch.sh

073876c

Remove loading of netcdf and cmake in module section. Not needed.

natalie-perlin reviewed Nov 10, 2022

View reviewed changes

Add scotch modulefile ito hpc-stack/modulefiles/compiler/compilerName…

f9dcf4b

…/compilerVersion for cases of serial installation.

ulmononian mentioned this pull request Nov 10, 2022

Add scotch library to EPIC's hpc-stack NOAA-EPIC/hpc-stack#7

Closed

ulmononian mentioned this pull request Nov 15, 2022

Add scotch library as a replacement for parmetis JCSDA/spack-stack#336

Closed

ulmononian and others added 4 commits November 17, 2022 20:33

Update Fortran and C compiler cmake flags in build_scotch.sh.

0037320

Merge branch 'NOAA-EMC:develop' into feature/add_scotch_hera

efb48eb

Merge branch 'NOAA-EMC:develop' into feature/add_scotch_hera

24f9771

Update build script.

9b2196f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add scotch library to hpc-stack #504

Add scotch library to hpc-stack #504

ulmononian commented Nov 9, 2022 •

edited

Loading

natalie-perlin Nov 10, 2022

natalie-perlin commented Nov 10, 2022

ulmononian commented Nov 10, 2022

edwardhartnett commented Nov 10, 2022

ulmononian commented Nov 10, 2022 •

edited

Loading

edwardhartnett commented Nov 10, 2022

ulmononian commented Nov 10, 2022 •

edited

Loading

aliabdolali commented Nov 10, 2022 •

edited

Loading

climbfuji commented Nov 10, 2022

ulmononian commented Nov 10, 2022 •

edited

Loading

edwardhartnett commented Nov 10, 2022

ulmononian commented Nov 10, 2022 •

edited

Loading

edwardhartnett commented Nov 10, 2022

ulmononian commented Nov 10, 2022 •

edited

Loading

ulmononian commented Nov 10, 2022

aliabdolali commented Nov 10, 2022

ulmononian commented Nov 10, 2022

ulmononian commented Nov 10, 2022

JessicaMeixner-NOAA commented Nov 10, 2022

ulmononian commented Nov 10, 2022

aliabdolali commented Nov 10, 2022

aerorahul commented Apr 28, 2023

natalie-perlin commented May 15, 2023

Add scotch library to hpc-stack #504

Are you sure you want to change the base?

Add scotch library to hpc-stack #504

Conversation

ulmononian commented Nov 9, 2022 • edited Loading

natalie-perlin Nov 10, 2022

Choose a reason for hiding this comment

natalie-perlin commented Nov 10, 2022

ulmononian commented Nov 10, 2022

edwardhartnett commented Nov 10, 2022

ulmononian commented Nov 10, 2022 • edited Loading

edwardhartnett commented Nov 10, 2022

ulmononian commented Nov 10, 2022 • edited Loading

aliabdolali commented Nov 10, 2022 • edited Loading

climbfuji commented Nov 10, 2022

ulmononian commented Nov 10, 2022 • edited Loading

edwardhartnett commented Nov 10, 2022

ulmononian commented Nov 10, 2022 • edited Loading

edwardhartnett commented Nov 10, 2022

ulmononian commented Nov 10, 2022 • edited Loading

ulmononian commented Nov 10, 2022

aliabdolali commented Nov 10, 2022

ulmononian commented Nov 10, 2022

ulmononian commented Nov 10, 2022

JessicaMeixner-NOAA commented Nov 10, 2022

ulmononian commented Nov 10, 2022

aliabdolali commented Nov 10, 2022

aerorahul commented Apr 28, 2023

natalie-perlin commented May 15, 2023

ulmononian commented Nov 9, 2022 •

edited

Loading

ulmononian commented Nov 10, 2022 •

edited

Loading

ulmononian commented Nov 10, 2022 •

edited

Loading

aliabdolali commented Nov 10, 2022 •

edited

Loading

ulmononian commented Nov 10, 2022 •

edited

Loading

ulmononian commented Nov 10, 2022 •

edited

Loading

ulmononian commented Nov 10, 2022 •

edited

Loading