Update docker container, add Docker auto-build and test, update documentation for building with spack-stack #466

mkavulich · 2024-05-01T22:08:30Z

This PR has several updates that improve SCM documentation, overhaul the docker container, updates new and existing tests, and adds modulefiles for building on MacOS and generic Linux.

Documentation

Documentation for this branch is built here: https://ccpp-scm-mkavulich.readthedocs.io/en/feature-macos_and_docker_build/

This update adds some information about testing to the Repository chapter (though more details should probably be included), and completely overhauls the Quick Start chapter.

Among the changes to the Quick Start chapter are:

Removing some less relevant text, and rearranging and updating some wording to improve flow of the chapter (this is supposed to be a "quick" start after all)
Replace non-standard section delimiters per Sphinx documentation
Include more information about spack-stack, including more links
Include instructions for loading environment on pre-configured platforms, as well as other procedures for building spack-stack or setting environment variables manually on other platforms such as MacOS and Linux
Update Python section for appropriate prerequisites
Use Sphinx built-in Notes markup to highlight important points
Update run commands, add more examples
Update Docker section with relevant changes (see next section)

Docker

The Dockerfile is updated to work with the latest CCPP code. The old "common community container" is no longer used, as it is no longer supported.
A github action is introduce that runs on each commit to main. This action will build the container and push it to Dockerhub under the tag "dtcenter/ccpp-scm:latest". This will ensure there is always an up-to-date, working container for the top of the main branch.

Testing

A new CI test will build and run the regressiont tests with the Docker container when a new pull request is created
Test ci_build_scm_ubuntu_22.04_nvidia.yml is modified to only run on pull_requests and when manually requested

Other

Add new "generic platform" modulefiles macos_clang and linux_gnu for use with spack-stack installations.

This PR will resolve #409 and partially resolve #455 (will still need a release container when the time comes)

mkavulich · 2024-05-01T22:57:07Z

Note: the new job to auto-push the latest container to Dockerhub makes use of repository secrets: I have provided my dockerhub credentials as DOCKERHUB_USERNAME and DOCKERHUB_TOKEN since I can write to the dtcenter organization on Dockerhub. If in the future I am not around, or if I lose access to that Dockerhub org, those secrets will have to be updated.

dustinswales

Thanks for making these changes.
I just wonder if we should rip out the NCEPlibs stuff now from the documentation, or wait until that PR gets into the physics?

scm/doc/TechGuide/chap_quick.rst

dustinswales · 2024-05-02T15:48:12Z

scm/doc/TechGuide/chap_quick.rst

+
+.. code:: bash
+
+ ./run_scm.py -f ../../test/rt_test_cases.py


Something we need to discuss are the SCM RTs.
rt_test_cases.py needs to be extended to

include the v7 tests, which we need to decide on. Same SDF/case combos as in v6 plus some new cases?
(Note we have a separate CI test that use the DEPHY repository, but there aren't any comparisons in that test, it just ensures the SCM works with the DEPHY repo. When I set this up I didn't do a deep evaluation on the case from the DEPHY repo, just more of a technical test that teh SCM could use the latest and greatest DEPHY cases. @bluefinweiwei has used the SCM with cases in the DEPHY repo for T&E, so I think it makes more sense to use a case that she is confident in. And then add a comparison step to the DEPHY CI test.
(Note We have a separate CI test for the UFS case generation. This test just creates the UFS case files from staged UFS RT data. It doesn't run the SCM with that case. We need to either expand this test to run the SCM and compare to baselines, OR leave this CI test as is and add a staged UFS case to the SCM, to be included in rt_test_cases.py?)

move the "supported v6" into the "unsupported" section. As long as these still work, we should include them in our testing suite.

Beef up the comparison step in the CI RT script to create plots of fields that differ, and upload these as a github artifact.

We only run the CI tests with GNU. Intel compilers aren't available on the guthub servers we do the CI, but we could create a Intel Docker container. TBH, for the RT CI test, we don't need to build the software stack as part of the test since ci_build_scm_ubuntu_22.04.yaml already does this, we just need to run the SCM and compare to baselines.

dustinswales · 2024-05-02T15:53:39Z

scm/doc/TechGuide/chap_intro.rst

@@ -103,7 +103,7 @@ This release bundle has some known limitations:

 -  As of this release, using the SCM over a land point with an LSM is
   possible through the use of UFS initial conditions (see 
-   :numref:`Section %s <UFS ICs>`). However, advective forcing terms
+   :numref:`Section %s <UFSreplay>`). However, advective forcing terms


FWIW. We are going to need to change all instances of "replay" in the documentation to "case generation".
There will be a subsequent PR with those code changes, so no need to address this at the moment.

This link is currently broken (not caught in the last PR) so this just updates the link to point to the correct place.

dustinswales · 2024-05-02T15:56:08Z

docker/Dockerfile

Should we name this file Dockerfile_GNU or Dockerfile_GNUv12? Just in case we get an Intel one soon.

Good point; do you think there is a possibility that Intel might be doable? Last time I worked with Docker (a few years ago) Intel was very difficult to get working in containers.

JCSDA uses Intel in Docker all the time. The only issue is that you can't distribute the containers to the public.

Ahh, okay that's good to know. If we do implement containers for testing we'll need a place to push them other than Dockerhub anyway, so it sounds like this may work.

mkavulich · 2024-05-17T22:19:17Z

@grantfirl Did you want a chance to review this or can I go ahead and merge it?

dustinswales

Looks good to me. A minor suggestion with the OS version in the CI scripts. And we will need to revisit documentation later anyways.

dustinswales · 2024-05-22T15:04:57Z

.github/workflows/build_and_push_docker_latest.yml

+
+jobs:
+  docker:
+    runs-on: ubuntu-latest


Maybe change to "ubuntu-22.04", the latest OS can change and cause hiccups.

Good point, updated

dustinswales · 2024-05-22T15:22:06Z

scm/doc/TechGuide/chap_quick.rst

We will need to revisit the documentation later anyways, at which point we can remove any deprecated dependencies.

docker/Dockerfile

grantfirl · 2024-05-22T16:57:07Z

scm/doc/TechGuide/chap_quick.rst

-pre-configured computing environment with a pre-compiled model. This is
-also an avenue for running this software with a Windows PC. See section
-:numref:`Section %s <docker>` for more information.
+SCM. We provide instructions on building the code from scratch (:numref:`Section %s <obtaining_code>`), as well as


Shortened because this is a "quick" start guide? Otherwise, this might be new information from someone unfamiliar with CCPP, etc.

Yes that's the idea, I wanted to trim down the less relevant bits in this chapter to make it a "quicker" start. The "Intro" chapter already covers this at a high level, I could expand that intro with this info about submodules specifically if you think that's needed.

grantfirl · 2024-05-22T16:59:28Z

scm/doc/TechGuide/chap_quick.rst

-For working with the development branches (stability not guaranteed),
-check out the ``main`` branch of the repository:
+Developers seeking to contribute code to the SCM or CCPP will need to use the most up-to-date
+version of the code, which can be found on the ``main`` branch of the repository: 


At one point, I think that the link contained basic information on git usage. Perhaps it isn't needed anymore. It also probably had links to our other docs, e.g. sci-doc and tech-doc. Those resources are still relevant for some SCM users, I'd imagine.

NVM. I see that it was just moved. Scrolling is hard. :)

grantfirl · 2024-05-22T17:03:00Z

scm/doc/TechGuide/chap_quick.rst

+needed for specifying the location of the various prerequisites. Users will need to set variables for the
+compilers (``CC``, ``CXX``, ``FC``), as well as the root directories for the library installs of NetCDF (``NetCDF_ROOT``),
+``bacio`` (``bacio_ROOT``), ``sp`` (``sp_ROOT``), and ``w3emc`` (``w3emc_ROOT``). This is the procedure used in the
+provided Dockerfile in ``ccpp-scm/docker/``, so users can reference that file for guidance on how to install this software


Ah, I see that you reference the "manual" approach used in the Dockerfile. That's fine. Disregard the comment above re: spack-stack.

grantfirl · 2024-05-22T17:05:15Z

scm/doc/TechGuide/chap_quick.rst

+``thompson_tables.tar.gz`` and ``MG_INCCN_data.tar.gz`` and extract
+to ``scm/data/physics_input_data/``.
+
+New with the SCM v7 release, static data is available for running cases with GOCART climatological aerosols (where the value of ``iaer`` in the ``&gfs_physics_nml`` namelist starts with 1; see the `CCPP Scientific Documentation <https://dtcenter.ucar.edu/GMTB/v6.0.0/sci_doc/_c_c_p_psuite_nml_desp.html>`__ for more information); one example of this is with the default namelist settings for the GFS_v17_HR3 scheme. This dataset is very large (~12 GB), so it is recommended only to download it if you will be using it.


grantfirl · 2024-05-22T17:07:35Z

scm/doc/TechGuide/chap_quick.rst

+     Windows users may need to omit the curly braces around environment variables: use ``$OUT_DIR``
+     instead of ``${OUT_DIR}``. 
+
+   For running through all supported cases and suites, use


The rt_test_cases.py has more than just the permutations of supported cases/suites, but that is probably not a big deal.

grantfirl · 2024-05-22T17:10:57Z

@mkavulich Can you update to the latest main for good measure before I press the merge button?

- Update Quick Start up through "Compilers" section - Update Repository chapter to include new "Testing" section (needs more expansion later) - Convert section marks to conform with Sphinx Documentation - Fix link and syntax errors

…te docs for latest python packages

…ome space specification

… docker CI

- remove a lot of fluff to get to the details quicker - Add example commands for installing spack-stack on MacOS - Update "multi-run" command for Docker Now updated through "4.3. Compiling SCM with CCPP"

…n this on pushes to main

…dulefile for flexibility

- Move MacOS example out to an external link (Github Discussion) - Move downloading of static data after build - Convert "NOTE" asides into proper Sphinx Note blocks - More details about debug mode - Remove yet another forum mention, how are these still out there!?

- Update SCM run commands, including some examples - Remove the deprecated "multirun" argument - Update argument descriptions - Update description of Docker software stack

… This job shouldn't need those credentials anyway now that I separated out the push step, so removing the Login step should fix this

…test" tag

mkavulich · 2024-05-22T18:06:53Z

@grantfirl Rebased my branch on the latest main

mkavulich requested review from dustinswales and grantfirl as code owners May 1, 2024 22:08

dustinswales approved these changes May 2, 2024

View reviewed changes

mkavulich mentioned this pull request May 2, 2024

Update procedure for building SCM for spack-stack, clean up existing build procedures #409

Closed

4 tasks

dustinswales approved these changes May 22, 2024

View reviewed changes

grantfirl reviewed May 22, 2024

View reviewed changes

docker/Dockerfile Show resolved Hide resolved

grantfirl reviewed May 22, 2024

View reviewed changes

docker/Dockerfile Show resolved Hide resolved

grantfirl reviewed May 22, 2024

View reviewed changes

grantfirl approved these changes May 22, 2024

View reviewed changes

mkavulich added 14 commits May 22, 2024 12:04

Add a macos clang module file

6e2c33a

Starting documentation updates for build/run

1f6014a

- Update Quick Start up through "Compilers" section - Update Repository chapter to include new "Testing" section (needs more expansion later) - Convert section marks to conform with Sphinx Documentation - Fix link and syntax errors

Update dockerfile for latest main branch

8d90db0

Update Dockerfile python setup; now runs test case successfully. Upda…

e0e18f7

…te docs for latest python packages

Testing out docker autobuild CI

d1d16a7

Don't run CI on all pushes

b6a1c31

Fixing docker context for CI

5c9b124

Small users guide addition about Docker usage, try to fix docker CI h…

6b12f31

…ome space specification

Testing docker CI

cc88611

Testing push of the test tag to dockerhub

edf6b18

Testing pull of docker CI test tag

f12b0b8

Testing HOME permissions for docker CI

048b018

Adding write permissions to $HOME for docker CI

d563524

Adding read+write permissions to $HOME recursively wqfor docker CI

9c8556f

mkavulich added 20 commits May 22, 2024 12:04

Put output in a subdirectory of $HOME to avoid permissions issues for…

2d0182e

… docker CI

chmod wasn't working as expected

a0c0c9f

chmod STILL wasn't working as expected

4f5d74d

I will never ever trust myself with a chmod command again

a9bda64

It works! Now to try running all test cases

4087a9f

Now to run it all together...

19350a5

Consolidate dockerfile RUN commands to shrink container size

e1ce0cc

More updates to Quick Start chapter:

78ad73c

- remove a lot of fluff to get to the details quicker - Add example commands for installing spack-stack on MacOS - Update "multi-run" command for Docker Now updated through "4.3. Compiling SCM with CCPP"

Split out the CI test that pushes new container to Dockerhub, only ru…

1ff40ce

…n this on pushes to main

Add Linux GNU modulefile, remove compiler version from macos_clang mo…

3d2ce1b

…dulefile for flexibility

Minor users guide updates

18fb473

Consolidate some repeated directory setup commands in Dockerfile

daf8563

More Quick Start updates

755e0c5

- Update SCM run commands, including some examples - Remove the deprecated "multirun" argument - Update argument descriptions - Update description of Docker software stack

Fix format error, convert some important notes to warnings

6cb6db8

Fix incorrect Dockerhub org name

1d2f9e7

Apparently GitHub secrets are not passed when PRs come in from forks.…

074fbd8

… This job shouldn't need those credentials anyway now that I separated out the push step, so removing the Login step should fix this

Fix dockerfile

2257062

Updates from Dustin's comments

0283c18

Run build and push docker test on ubuntu-22.04 per concerns about "la…

10c8f05

…test" tag

mkavulich force-pushed the feature/macos_and_docker_build branch from 97ee4b3 to 10c8f05 Compare May 22, 2024 18:06

grantfirl merged commit 046559b into NCAR:main May 22, 2024
18 of 22 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update docker container, add Docker auto-build and test, update documentation for building with spack-stack #466

Update docker container, add Docker auto-build and test, update documentation for building with spack-stack #466

mkavulich commented May 1, 2024

mkavulich commented May 1, 2024

dustinswales left a comment

dustinswales May 2, 2024

dustinswales May 2, 2024

mkavulich May 2, 2024

dustinswales May 2, 2024

mkavulich May 2, 2024

climbfuji May 2, 2024

mkavulich May 2, 2024

mkavulich commented May 17, 2024

dustinswales left a comment

dustinswales May 22, 2024

mkavulich May 22, 2024

dustinswales May 22, 2024

grantfirl May 22, 2024

mkavulich May 22, 2024

grantfirl May 22, 2024

grantfirl May 22, 2024

grantfirl May 22, 2024

grantfirl May 22, 2024

grantfirl May 22, 2024

grantfirl commented May 22, 2024

mkavulich commented May 22, 2024

Update docker container, add Docker auto-build and test, update documentation for building with spack-stack #466

Update docker container, add Docker auto-build and test, update documentation for building with spack-stack #466

Conversation

mkavulich commented May 1, 2024

Documentation

Docker

Testing

Other

mkavulich commented May 1, 2024

dustinswales left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mkavulich commented May 17, 2024

dustinswales left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

grantfirl commented May 22, 2024

mkavulich commented May 22, 2024