Print LmodError when loading GCCcore-12.2.0-based modules on `zen4` #841

casparvl · 2024-12-10T17:17:47Z

Depends on: Add known issue on foss-2022b not being supported for zen4 docs#357

Implements the idea from https://gitlab.com/eessi/support/-/issues/37#note_2159031831

But, not currently working, because the first module that gets installed that uses GCCcore-12.2.0 as dependency will try to load it (even with --module-only), which then fails:

== creating module...
  >> generating module file @ /home/casparl/eessi/versions/2023.06/software/linux/x86_64/amd/zen4/modules/all/GCCcore/12.2.0.lua
== ... (took < 1 sec)
== permissions [skipped]
== packaging [skipped]
  >> running command:
        [started at: 2024-12-10 18:12:47]
        [working dir: /gpfs/home4/casparl/eessi/versions/2023.06/software/linux/x86_64/amd/zen4/modules]
        [output logged in /scratch-local/casparl.8987353/eb-bscq9hvh/easybuild-run_cmd-pxom1tke.log]
        bzip2 /home/casparl/eessi/versions/2023.06/software/linux/x86_64/amd/zen4/software/GCCcore/12.2.0/easybuild/easybuild-GCCcore-12.2.0-20241210.181247.log
  >> command completed: exit 0, ran in < 1s
== COMPLETED: Installation ended successfully (took 1 secs)
== Results of the build can be found in the log file(s) /home/casparl/eessi/versions/2023.06/software/linux/x86_64/amd/zen4/software/GCCcore/12.2.0/easybuild/easybuild-GCCcore-12.2.0-20241210.181247.log.bz2
== processing EasyBuild easyconfig /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen4/software/EasyBuild/4.9.4/easybuild/easyconfigs/p/pkgconf/pkgconf-1.9.3-GCCcore-12.2.0.eb
== building and installing pkgconf/1.9.3-GCCcore-12.2.0...
  >> installation prefix: /home/casparl/eessi/versions/2023.06/software/linux/x86_64/amd/zen4/software/pkgconf/1.9.3-GCCcore-12.2.0
== fetching files [skipped]
== creating build dir, resetting environment...
  >> build dir: /tmp/casparl/easybuild/build/pkgconf/1.9.3/GCCcore-12.2.0
== Running post-ready hook...
== ... (took < 1 sec)
== unpacking [skipped]
== patching [skipped]
== preparing...
== Running pre-prepare hook...
== ... (took < 1 sec)
== FAILED: Installation ended unsuccessfully (build directory: /tmp/casparl/easybuild/build/pkgconf/1.9.3/GCCcore-12.2.0): build failed (first 300 chars): Module command '/usr/share/lmod/lmod/libexec/lmod python show GCCcore/12.2.0' failed with exit code 1; stderr: Lmod has detected
the following error: Unable to load module because of error when evaluating modulefile:
     /home/casparl/eessi/versions/2023.06/software/linux/x86_64/amd/zen4/modules/al (took 0 secs)
== Results of the build can be found in the log file(s) /scratch-local/casparl.8987353/eb-bscq9hvh/easybuild-pkgconf-1.9.3-20241210.181248.DZrFb.log
0:00:02  1 out of 37 easyconfigs done: GCCcore/12.2.0 (OK)
ERROR: Build of /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen4/software/EasyBuild/4.9.4/easybuild/easyconfigs/p/pkgconf/pkgconf-1.9.3-GCCcore-12.2.0.eb failed (err: "build failed (first 300 chars): Module command '/usr/share/lmod/lmod/libexec/lmod python show GCCcore/12.2.0' failed with exit code 1; stderr: Lmod has detected the following error: Unable to load module because of error when evaluating modulefile:\n     /home/casparl/eessi/versions/2023.06/software/linux/x86_64/amd/zen4/modules/al")

maybe we can make that pre-prepare hook do nothing. Or skip the prepare phase. NOt sure...

…dError when they are loaded

eessi-bot · 2024-12-10T17:17:51Z

Instance eessi-bot-mc-aws is configured to build for:

architectures: x86_64/generic, x86_64/intel/haswell, x86_64/intel/skylake_avx512, x86_64/amd/zen2, x86_64/amd/zen3, aarch64/generic, aarch64/neoverse_n1, aarch64/neoverse_v1
repositories: eessi.io-2023.06-compat, eessi-hpc.org-2023.06-software, eessi-hpc.org-2023.06-compat, eessi.io-2023.06-software

riscv-eessi-io-bot · 2024-12-10T17:17:51Z

Instance eessi-bot-riscv is configured to build for:

architectures: riscv64/generic
repositories: riscv.eessi.io-20240402

eessi-bot · 2024-12-10T17:17:52Z

Instance eessi-bot-mc-azure is configured to build for:

architectures: x86_64/amd/zen4
repositories: eessi.io-2023.06-compat, eessi.io-2023.06-software

eb_hooks.py

…e of the step-hooks, so we can unset it at the end

eb_hooks.py

casparvl · 2024-12-11T14:48:15Z

Ok, this PR is more or less ready, but we should create a known issues page on the zen4 tree missing 2022b / GCCcore 12.2.0. I think it can be very simple and state that because of issues observed with the OpenBLAS from that toolchain generation, we decided not to support it. It also makes sense: zen4 was only release end of 2022, so the 2022b stack would have had very little support for it.

casparvl · 2024-12-11T16:58:49Z

Ok, I'll need EESSI/docs#357 to be merged first. Then, I'll put a link to that part of the docs in the LmodError.

ocaisa · 2024-12-12T11:10:28Z

eb_hooks.py

+EESSI_IGNORE_ZEN4_GCC1220_ENVVAR="EESSI_IGNORE_LMOD_ERROR_ZEN4_GCC1220"
+
+def is_gcccore_1220_based(ecname, ecversion, tcname, tcversion):
+    """Checks if this easyconfig either _is_ or _uses_ a GCCcore-12.2.2 based toolchain"""


Suggested change

"""Checks if this easyconfig either _is_ or _uses_ a GCCcore-12.2.2 based toolchain"""

"""Checks if this easyconfig either _is_ or _uses_ a GCCcore-12.2.0 based toolchain"""

ocaisa · 2024-12-12T11:12:27Z

eb_hooks.py

+        # Need to escape newline character so that the newline character actually ends up in the module file
+        # (otherwise, it splits the string, and a 2-line string ends up in the modulefile, resulting in syntax error)
+        errmsg = "EasyConfigs using toolchains based on GCCcore-12.2.0 are not supported for the Zen4 architecture.\\n"
+        errmsg += "See https://www.eessi.io/docs/known_issues/eessi-2023.06/"


Suggested change

errmsg += "See https://www.eessi.io/docs/known_issues/eessi-2023.06/"

errmsg += "See https://www.eessi.io/docs/known_issues/eessi-2023.06/#gcc-1220-and-foss-2022b-based-modules-cannot-be-loaded-on-zen4-architecture"

ocaisa · 2024-12-12T11:20:13Z

eb_hooks.py

+# Make sure a single environment variable name is used for this throughout the hooks
+EESSI_IGNORE_ZEN4_GCC1220_ENVVAR="EESSI_IGNORE_LMOD_ERROR_ZEN4_GCC1220"
+
+def is_gcccore_1220_based(ecname, ecversion, tcname, tcversion):


This feels brittle, there's no sanity checking on the arguments. If you give the arguments in the wrong order, the function will happily proceed. Don't particularly want you to bend over backwards to check the arguments, I think kwargs with default None would at least be clear and less error prone.

ocaisa · 2024-12-12T11:23:00Z

eb_hooks.py

+                                 EESSI_FORCE_ATTR)
+
+
+# We do this as early as possible - and remove it all the way in the last step hook (post_testcases_hook)


This comment is no longer accurate (I think)

ocaisa · 2024-12-12T11:27:22Z

eb_hooks.py

+
+
+# We do this as early as possible - and remove it all the way in the last step hook (post_testcases_hook)
+def pre_prepare_hook_ignore_zen4_gcccore1220_error(self, *args, **kwargs):


Is there really any harm in collapsing this into your pre_fetch hook? It would be nice to keep the setting and unsetting unified for our future selves to understand better what would need to be changed.

Or is the environment variable not persistent?

ocaisa · 2024-12-12T11:28:10Z

eb_hooks.py

+        os.environ[EESSI_IGNORE_ZEN4_GCC1220_ENVVAR] = "1"
+
+
+def post_prepare_hook_ignore_zen4_gcccore1220_error(self, *args, **kwargs):


Same here, is there any harm to just adding this to post_module_hook?

Initial attempt for an EasyBuild hook that makes modules print an Lmo…

d61f33d

…dError when they are loaded

ocaisa reviewed Dec 11, 2024

View reviewed changes

eb_hooks.py Outdated Show resolved Hide resolved

ocaisa reviewed Dec 11, 2024

View reviewed changes

eb_hooks.py Outdated Show resolved Hide resolved

This works, but we should probably set the environment variable in on…

9bb52dc

…e of the step-hooks, so we can unset it at the end

ocaisa reviewed Dec 11, 2024

View reviewed changes

eb_hooks.py Outdated Show resolved Hide resolved

Only set env var for prepare step

c26a1e5

ocaisa reviewed Dec 11, 2024

View reviewed changes

eb_hooks.py Show resolved Hide resolved

eb_hooks.py Show resolved Hide resolved

Caspar van Leeuwen added 3 commits December 11, 2024 14:28

Remove unused code

600adee

Use step hooks to set and unset force and module-only

b3fade9

Rename hooks to make more sense

6a1700e

casparvl marked this pull request as ready for review December 11, 2024 16:57

casparvl marked this pull request as draft December 11, 2024 16:58

ocaisa requested changes Dec 12, 2024

View reviewed changes

ocaisa reviewed Dec 12, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Print LmodError when loading GCCcore-12.2.0-based modules on `zen4` #841

Print LmodError when loading GCCcore-12.2.0-based modules on `zen4` #841

casparvl commented Dec 10, 2024 •

edited by ocaisa

Loading

eessi-bot bot commented Dec 10, 2024

riscv-eessi-io-bot bot commented Dec 10, 2024

eessi-bot bot commented Dec 10, 2024

casparvl commented Dec 11, 2024 •

edited

Loading

casparvl commented Dec 11, 2024

ocaisa Dec 12, 2024

ocaisa Dec 12, 2024

ocaisa Dec 12, 2024

ocaisa Dec 12, 2024

ocaisa Dec 12, 2024

ocaisa Dec 12, 2024

ocaisa Dec 12, 2024

	"""Checks if this easyconfig either _is_ or _uses_ a GCCcore-12.2.2 based toolchain"""
	"""Checks if this easyconfig either _is_ or _uses_ a GCCcore-12.2.0 based toolchain"""

	errmsg += "See https://www.eessi.io/docs/known_issues/eessi-2023.06/"
	errmsg += "See https://www.eessi.io/docs/known_issues/eessi-2023.06/#gcc-1220-and-foss-2022b-based-modules-cannot-be-loaded-on-zen4-architecture"

		EESSI_FORCE_ATTR)


		# We do this as early as possible - and remove it all the way in the last step hook (post_testcases_hook)



		# We do this as early as possible - and remove it all the way in the last step hook (post_testcases_hook)
		def pre_prepare_hook_ignore_zen4_gcccore1220_error(self, args, *kwargs):

		os.environ[EESSI_IGNORE_ZEN4_GCC1220_ENVVAR] = "1"


		def post_prepare_hook_ignore_zen4_gcccore1220_error(self, args, *kwargs):

Print LmodError when loading GCCcore-12.2.0-based modules on zen4 #841

Are you sure you want to change the base?

Print LmodError when loading GCCcore-12.2.0-based modules on zen4 #841

Conversation

casparvl commented Dec 10, 2024 • edited by ocaisa Loading

eessi-bot bot commented Dec 10, 2024

riscv-eessi-io-bot bot commented Dec 10, 2024

eessi-bot bot commented Dec 10, 2024

casparvl commented Dec 11, 2024 • edited Loading

casparvl commented Dec 11, 2024

ocaisa Dec 12, 2024

Choose a reason for hiding this comment

ocaisa Dec 12, 2024

Choose a reason for hiding this comment

ocaisa Dec 12, 2024

Choose a reason for hiding this comment

ocaisa Dec 12, 2024

Choose a reason for hiding this comment

ocaisa Dec 12, 2024

Choose a reason for hiding this comment

ocaisa Dec 12, 2024

Choose a reason for hiding this comment

ocaisa Dec 12, 2024

Choose a reason for hiding this comment

Print LmodError when loading GCCcore-12.2.0-based modules on `zen4` #841

Print LmodError when loading GCCcore-12.2.0-based modules on `zen4` #841

casparvl commented Dec 10, 2024 •

edited by ocaisa

Loading

casparvl commented Dec 11, 2024 •

edited

Loading