add spack sanity check for e4s/23.05

[shahzebsiddiqui]

@jscook2345

This MR is meant to address #200

I ran this test locally, the csh one doesn't work because there is a bug in buildtest, but the ones in bash work

(buildtest) siddiq90@login05> buildtest bd -b perlmutter_spack_check.yml
╭────────────────────────────────────────────── buildtest summary ───────────────────────────────────────────────╮
│                                                                                                                │
│ User:               siddiq90                                                                                   │
│ Hostname:           login05                                                                                    │
│ Platform:           Linux                                                                                      │
│ Current Time:       2023/11/20 08:50:40                                                                        │
│ buildtest path:     /global/homes/s/siddiq90/gitrepos/buildtest/bin/buildtest                                  │
│ buildtest version:  1.7                                                                                        │
│ python path:        /global/u1/s/siddiq90/.local/share/virtualenvs/buildtest-WqshQcL1/bin/python3              │
│ python version:     3.9.7                                                                                      │
│ Configuration File: /global/u1/s/siddiq90/gitrepos/buildtest-nersc/config.yml                                  │
│ Test Directory:     /global/u1/s/siddiq90/gitrepos/buildtest/var/tests                                         │
│ Report File:        /global/u1/s/siddiq90/gitrepos/buildtest/var/report.json                                   │
│ Command:            /global/homes/s/siddiq90/gitrepos/buildtest/bin/buildtest bd -b perlmutter_spack_check.yml │
│                                                                                                                │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────  Discovering Buildspecs ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
                                       Discovered buildspecs
╔═════════════════════════════════════════════════════════════════════════════════════════════════╗
║ buildspec                                                                                       ║
╟─────────────────────────────────────────────────────────────────────────────────────────────────╢
║ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/apps/spack/perlmutter_spack_check.yml ║
╚═════════════════════════════════════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Parsing Buildspecs ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
spack_e4s_22.11: skipping test due to 'skip' property.
Valid Buildspecs: 1
Invalid Buildspecs: 0
/global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/apps/spack/perlmutter_spack_check.yml: VALID
Total builder objects created: 2
                                                                                            Builders by type=script
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ builder                         ┃ type   ┃ executor              ┃ compiler ┃ nodes ┃ procs ┃ description ┃ buildspecs                                                                                      ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ spack_sanity_e4s_23.05/65939961 │ script │ perlmutter.local.bash │ None     │ None  │ None  │             │ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/apps/spack/perlmutter_spack_check.yml │
├─────────────────────────────────┼────────┼───────────────────────┼──────────┼───────┼───────┼─────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────┤
│ spack_sanity_e4s_23.05/0f3fb2ac │ script │ perlmutter.local.csh  │ None     │ None  │ None  │             │ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/apps/spack/perlmutter_spack_check.yml │
└─────────────────────────────────┴────────┴───────────────────────┴──────────┴───────┴───────┴─────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────┘
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Building Test ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
spack_sanity_e4s_23.05/65939961: Creating test directory: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.local.bash/perlmutter_spack_check/spack_sanity_e4s_23.05/65939961
spack_sanity_e4s_23.05/65939961: Creating the stage directory: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.local.bash/perlmutter_spack_check/spack_sanity_e4s_23.05/65939961/stage
spack_sanity_e4s_23.05/65939961: Writing build script: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.local.bash/perlmutter_spack_check/spack_sanity_e4s_23.05/65939961/spack_sanity_e4s_23.05_build.sh
spack_sanity_e4s_23.05/0f3fb2ac: Creating test directory: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.local.csh/perlmutter_spack_check/spack_sanity_e4s_23.05/0f3fb2ac
spack_sanity_e4s_23.05/0f3fb2ac: Creating the stage directory: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.local.csh/perlmutter_spack_check/spack_sanity_e4s_23.05/0f3fb2ac/stage
spack_sanity_e4s_23.05/0f3fb2ac: Writing build script: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.local.csh/perlmutter_spack_check/spack_sanity_e4s_23.05/0f3fb2ac/spack_sanity_e4s_23.05_build.sh
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Running Tests ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Spawning 8 processes for processing builders
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Iteration 1 ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
spack_sanity_e4s_23.05/0f3fb2ac does not have any dependencies adding test to queue
spack_sanity_e4s_23.05/65939961 does not have any dependencies adding test to queue
     Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder                         ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ spack_sanity_e4s_23.05/65939961 │
│ spack_sanity_e4s_23.05/0f3fb2ac │
└─────────────────────────────────┘
spack_sanity_e4s_23.05/0f3fb2ac: Current Working Directory : /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.local.csh/perlmutter_spack_check/spack_sanity_e4s_23.05/0f3fb2ac/stage
spack_sanity_e4s_23.05/65939961: Current Working Directory : /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.local.bash/perlmutter_spack_check/spack_sanity_e4s_23.05/65939961/stage
spack_sanity_e4s_23.05/0f3fb2ac: Running Test via command: csh -ef spack_sanity_e4s_23.05_build.sh
spack_sanity_e4s_23.05/65939961: Running Test via command: bash --norc --noprofile -eo pipefail spack_sanity_e4s_23.05_build.sh
spack_sanity_e4s_23.05/0f3fb2ac failed to submit job with returncode: 1
───────────────────────────────────────────────────────────────────────────────────────────────────────────── Error Message for spack_sanity_e4s_23.05/0f3fb2ac ──────────────────────────────────────────────────────────────────────────────────────────────────────────────
Badly placed ()'s.

spack_sanity_e4s_23.05/0f3fb2ac: Detected failure in running test, will attempt to retry test: 1 times
spack_sanity_e4s_23.05/0f3fb2ac: Run - 1/1
spack_sanity_e4s_23.05/0f3fb2ac: Running Test via command: csh -ef spack_sanity_e4s_23.05_build.sh
spack_sanity_e4s_23.05/0f3fb2ac: failed to submit job with returncode: 1
spack_sanity_e4s_23.05/0f3fb2ac: Test completed in 0.043453 seconds
spack_sanity_e4s_23.05/0f3fb2ac: Test completed with returncode: 1
spack_sanity_e4s_23.05/0f3fb2ac: Writing output file -  /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.local.csh/perlmutter_spack_check/spack_sanity_e4s_23.05/0f3fb2ac/spack_sanity_e4s_23.05.out
spack_sanity_e4s_23.05/0f3fb2ac: Writing error file - /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.local.csh/perlmutter_spack_check/spack_sanity_e4s_23.05/0f3fb2ac/spack_sanity_e4s_23.05.err
spack_sanity_e4s_23.05/65939961: Test completed in 10.075249 seconds
spack_sanity_e4s_23.05/65939961: Test completed with returncode: 0
spack_sanity_e4s_23.05/65939961: Writing output file -  /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.local.bash/perlmutter_spack_check/spack_sanity_e4s_23.05/65939961/spack_sanity_e4s_23.05.out
spack_sanity_e4s_23.05/65939961: Writing error file - /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.local.bash/perlmutter_spack_check/spack_sanity_e4s_23.05/65939961/spack_sanity_e4s_23.05.err
                                                           Test Summary
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ builder                         ┃ executor              ┃ status ┃ checks (ReturnCode, Regex, Runtime) ┃ returncode ┃ runtime   ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ spack_sanity_e4s_23.05/65939961 │ perlmutter.local.bash │ PASS   │ None None None                      │ 0          │ 10.075249 │
├─────────────────────────────────┼───────────────────────┼────────┼─────────────────────────────────────┼────────────┼───────────┤
│ spack_sanity_e4s_23.05/0f3fb2ac │ perlmutter.local.csh  │ FAIL   │ None None None                      │ 1          │ 0.043453  │
└─────────────────────────────────┴───────────────────────┴────────┴─────────────────────────────────────┴────────────┴───────────┘



Passed Tests: 1/2 Percentage: 50.000%
Failed Tests: 1/2 Percentage: 50.000%


Adding 2 test results to /global/u1/s/siddiq90/gitrepos/buildtest/var/report.json
Writing Logfile to: /global/u1/s/siddiq90/gitrepos/buildtest/var/logs/buildtest_2za662eg.log

The output of the test that passed is the following

(buildtest) siddiq90@login05> buildtest it query -o -e spack_sanity_e4s_23.05/65939961
──────────────────────────────────────────────────────────────────────────────────────────────────────── spack_sanity_e4s_23.05/65939961-d74d-4f2b-a658-9c8619e6db14 ─────────────────────────────────────────────────────────────────────────────────────────────────────────
Executor: perlmutter.local.bash
Description:
State: PASS
Returncode: 0
Runtime: 10.075249 sec
Starttime: 2023/11/20 08:50:40
Endtime: 2023/11/20 08:50:50
Command: bash --norc --noprofile -eo pipefail spack_sanity_e4s_23.05_build.sh
Test Script: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.local.bash/perlmutter_spack_check/spack_sanity_e4s_23.05/65939961/spack_sanity_e4s_23.05.sh
Build Script: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.local.bash/perlmutter_spack_check/spack_sanity_e4s_23.05/65939961/spack_sanity_e4s_23.05_build.sh
Output File: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.local.bash/perlmutter_spack_check/spack_sanity_e4s_23.05/65939961/spack_sanity_e4s_23.05.out
Error File: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.local.bash/perlmutter_spack_check/spack_sanity_e4s_23.05/65939961/spack_sanity_e4s_23.05.err
Log File: /global/u1/s/siddiq90/gitrepos/buildtest/var/logs/buildtest_2za662eg.log
────────────────────────────────────────────────── Output File: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.local.bash/perlmutter_spack_check/spack_sanity_e4s_23.05/65939961/spack_sanity_e4s_23.05.out ───────────────────────────────────────────────────
    cce
    cuda
    data
    gcc
    math-libs
    nvhpc
    tools
444
38
174
47
59
85
58

─────────────────────────────────────────────────── Error File: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.local.bash/perlmutter_spack_check/spack_sanity_e4s_23.05/65939961/spack_sanity_e4s_23.05.err ───────────────────────────────────────────────────
    _______________________________________________________________________________________________________
     The Extreme-Scale Scientific Software Stack (E4S) is accessible via the Spack package manager.
     In order to access the production stack, you will need to load a spack environment. Here are some tips to get started:
     'spack env list' - List all Spack environments
     'spack env activate gcc' - Activate the "gcc" Spack environment
     'spack env status' - Display the active Spack environment
     'spack load amrex' - Load the "amrex" Spack package into your user environment
     For additional support, please refer to the following references:
       NERSC E4S Documentation: https://docs.nersc.gov/applications/e4s/
       E4S Documentation: https://e4s.readthedocs.io
       Spack Documentation: https://spack.readthedocs.io/en/latest/
       Spack Slack: https://spackpm.slack.com
    ______________________________________________________________________________________________________