Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to use shared scotch libs #1024

Draft
wants to merge 6 commits into
base: develop
Choose a base branch
from

Conversation

ulmononian
Copy link

@ulmononian ulmononian commented Jun 20, 2023

Pull Request Summary

Enable the (optional) use of shared scotch libraries.

Description

Some changes to cmake/FindSCOTCH.cmake and model/src/CMakeLists.txt are added here so that shared scotch libraries can be used when appropriate/wanted. Future spack-stack installations will most likely install only the shared libraries, so the capability will become necessary. Static library options need to be retain for operational machines where shared libraries are not supported.

Please also include the following information:
Potential reviewers: @MatthewMasarik-NOAA @JessicaMeixner-NOAA

  • enhancement
  • Not sure if it will change results

Issue(s) addressed

Fixes #1021

Commit Message

Enable the (optional) use of shared scotch libraries.

Check list

Testing

  • How were these changes tested?
    using the PDLIB RT cmake settings from the UFS-WM (i.e. cmake -DAPP=S2SW -DCCPP_SUITES=FV3_GFS_v17_coupled_p8 -DPDLIB=ON ..)
  • Are the changes covered by regression tests? (If not, why? Do new tests need to be added?)
    I am only sure of how they would be covered in the UFS-WM S2SW/S2SWA configurations where PDLIB is turned on.
  • Have the matrix regression tests been run (if yes, please note HPC and compiler)?
    No
  • Please indicate the expected changes in the regression test output, (Note the list of known non-identical tests.)
  • Please provide the summary output of matrix.comp (matrix.Diff.txt, matrixCompFull.txt and matrixCompSummary.txt):

@ulmononian ulmononian marked this pull request as ready for review June 22, 2023 16:06
@ulmononian
Copy link
Author

ulmononian commented Jun 22, 2023

this was tested within the ufs-wm compilation stage using the spack-stack/1.4.0 scotch/7.0.3 installation (note this is a static set of scotch libraries). the option to set USE_SHARED_SCOTCH as a cmake arg. was added, so that static or shared can be specified in the cmake command, i.e.:

cmake -DAPP=S2SW -DCCPP_SUITES=FV3_GFS_v17_coupled_p8 -DPDLIB=ON -DUSE_SHARED_SCOTCH=[ON/OFF] ..

i tested it with USE_SHARED_SCOTCH both ON and OFF and the behavior was as expected, i.e., when set to ON, it finds the static scotch libs and proceeds to finish the cmake step; when OFF, it throws an error letting the user know that the requested (shared) libs are not present.

Copy link
Collaborator

@MatthewMasarik-NOAA MatthewMasarik-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ulmononian has this been tested by anyone beyond the compilation stage? I've found it's possible for scotch and WW3 to build without error, but crash at runtime when it enters the scotch call. Please confirm the functionality works at runtime as well.

@ulmononian
Copy link
Author

@MatthewMasarik-NOAA it has been tested in compilation and in ufs-wm rts. it does not change the outcome of the ufs-wm rt's where PDLIB=ON; that is, they still fail in the scotch call. basically, testing this modification in the wm only checks if the USE_SHARED_FLAG is working, and as we only have static scotch libs. installed, it just finds those.

if someone on the ww3 team could test these changes in the internal ww3 tests, i would appreciate it. i am happy to try and run them myself, but would need instruction.

@MatthewMasarik-NOAA
Copy link
Collaborator

@MatthewMasarik-NOAA it has been tested in compilation and in ufs-wm rts. it does not change the outcome of the ufs-wm rt's where PDLIB=ON; that is, they still fail in the scotch call. basically, testing this modification in the wm only checks if the USE_SHARED_FLAG is working, and as we only have static scotch libs. installed, it just finds those.

if someone on the ww3 team could test these changes in the internal ww3 tests, i would appreciate it. i am happy to try and run them myself, but would need instruction.

@ulmononian is this in reference to intel or gnu spack-stack? Previously I verified the intel spack-stack SCOTCH install built and ran successfully (see comment, JCSDA/spack-stack#550 (comment)). If the current static intel spack-stack version is failing to run, then that's news to me. I would suggest resolving that issue before moving on to building with shared.

@ulmononian
Copy link
Author

@MatthewMasarik-NOAA sorry -- you are correct that only the gnu spack-stack scotch installation is failing.

this pr is just to add the option for using shared libraries in the future. we won't switch from static to shared on the spack side until discussing with your team & as you said, resolving the static (gnu) issues first.

@MatthewMasarik-NOAA
Copy link
Collaborator

@MatthewMasarik-NOAA sorry -- you are correct that only the gnu spack-stack scotch installation is failing.

this pr is just to add the option for using shared libraries in the future. we won't switch from static to shared on the spack side until discussing with your team & as you said, resolving the static (gnu) issues first.

@ulmononian, okay glad to hear the intel spack-stack is working.

I understand this is just an option. Though if we add the ability to select an option, the functionality it selects should be working, right? If not, doesn't this just kick the same problem down the road?

@MatthewMasarik-NOAA
Copy link
Collaborator

Hi @ulmononian, I'll mention that when we were trying to resolve scotch issues on orion sometime ago, I went down a similar path trying to build SCOTCH/WW3 with spack (not spack-stack, though). It built SCOTCH default as .so files, so I made similar changes to FindSCOTCH.cmake to use those instead of .a. It seemed it was able to build without reporting an error, but when run it crashes in initialization where execution hits the SCOTCH call. I wasn't successful and that was my only attempt at using shared instead of static files. I might suggest seeing if you can build/run WW3 from SCOTCH .so files using hpc-stack first, if you can do that, try with spack-stack.

As far as testing, I think you are going about it the easiest way, using the a coupled UFS RT. I'm happy to give you a demo on how you could debug in WW3 standalone though, if that's something you'd like to try. Whichever of those you use you will always have to rebuild SCOTCH + WW3, and then do a run as well, at least up to model init, because that is the only way you can confirm a successful build. Since the builds and run details like job card resources are all hidden when you use rt.sh in UFS, I think it's going to be easiest continuing with testing like that, vs. standalone WW3, but I will teach you if you like.

@MatthewMasarik-NOAA
Copy link
Collaborator

@ulmononian we've merged in a fix for the CI. Please sync when you have a chance

@MatthewMasarik-NOAA
Copy link
Collaborator

Hi @ulmononian, there hasn't been any activity on this in awhile. Do you plan to pursue in the near future, or should this be a draft for the time being? Thanks for any status update

@ulmononian
Copy link
Author

@MatthewMasarik-NOAA sorry about the radio silence. this was removed from my radar. i don't think i'll be able to pursue this for a while, so it can be kept as a draft (or closed).

@MatthewMasarik-NOAA
Copy link
Collaborator

@MatthewMasarik-NOAA sorry about the radio silence. this was removed from my radar. i don't think i'll be able to pursue this for a while, so it can be kept as a draft (or closed).

Hi @ulmononian, no worries. thanks for letting me know where things stand. If spack-stack does go to just shared builds then this will be nice to have, and I suspect it is already pretty close. So I think we should keep it in the queue for the time being. It looks like as a maintainer I'm not able to mark the PR as a draft, could you mark it a draft? thanks again for the update.

@JessicaMeixner-NOAA JessicaMeixner-NOAA marked this pull request as draft November 2, 2023 11:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add shared library option to FindSCOTCH.cmake
2 participants