-
Notifications
You must be signed in to change notification settings - Fork 203
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add Generation module naming scheme #3547
base: develop
Are you sure you want to change the base?
Changes from 3 commits
b762779
3477e9b
7f5a05d
6b93ea6
fd974da
009ca00
85bed5a
e56e0e5
0066a71
1c16860
9df685d
1b4392f
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,115 @@ | ||
## | ||
# Copyright 2016-2021 Ghent University | ||
# | ||
# This file is part of EasyBuild, | ||
# originally created by the HPC team of Ghent University (http://ugent.be/hpc/en), | ||
# with support of Ghent University (http://ugent.be/hpc), | ||
# the Flemish Supercomputer Centre (VSC) (https://www.vscentrum.be), | ||
# Flemish Research Foundation (FWO) (http://www.fwo.be/en) | ||
# and the Department of Economy, Science and Innovation (EWI) (http://www.ewi-vlaanderen.be/en). | ||
# | ||
# https://github.com/easybuilders/easybuild | ||
# | ||
# EasyBuild is free software: you can redistribute it and/or modify | ||
# it under the terms of the GNU General Public License as published by | ||
# the Free Software Foundation v2. | ||
# | ||
# EasyBuild is distributed in the hope that it will be useful, | ||
# but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
# GNU General Public License for more details. | ||
# | ||
# You should have received a copy of the GNU General Public License | ||
# along with EasyBuild. If not, see <http://www.gnu.org/licenses/>. | ||
## | ||
""" | ||
Implementation of a different generation specific module naming scheme using release dates. | ||
:author: Thomas Eylenbosch (Gluo N.V.) | ||
""" | ||
|
||
import os | ||
|
||
from easybuild.tools.module_naming_scheme.mns import ModuleNamingScheme | ||
# from easybuild.tools.module_naming_scheme.utilities import det_full_ec_version | ||
|
||
DUMMY_TOOLCHAIN_NAME = 'dummy' | ||
DUMMY_TOOLCHAIN_VERSION = 'dummy' | ||
|
||
SYSTEM_TOOLCHAIN_NAME = 'system' | ||
|
||
|
||
class GenerationModuleNamingScheme(ModuleNamingScheme): | ||
"""Class implementing the categorized module naming scheme.""" | ||
|
||
REQUIRED_KEYS = ['name', 'version', 'versionsuffix', 'toolchain'] | ||
|
||
def det_full_module_name(self, ec): | ||
""" | ||
Determine short module name, i.e. the name under which modules will be exposed to users. | ||
Examples: GCC/4.8.3, OpenMPI/1.6.5, OpenBLAS/0.2.9, HPL/2.1, Python/2.7.5 | ||
""" | ||
|
||
return os.path.join(self.det_module_subdir(ec), self.det_short_module_name(ec)) | ||
|
||
def det_short_module_name(self, ec): | ||
""" | ||
Determine short module name, i.e. the name under which modules will be exposed to users. | ||
Examples: GCC/4.8.3, OpenMPI/1.6.5, OpenBLAS/0.2.9, HPL/2.1, Python/2.7.5 | ||
""" | ||
return os.path.join(ec['name'], self.det_full_version(ec)) | ||
|
||
def det_full_version(self, ec): | ||
"""Determine full version, taking into account version prefix/suffix.""" | ||
# versionprefix is not always available (e.g., for toolchains) | ||
versionprefix = ec.get('versionprefix', '') | ||
return versionprefix + ec['version'] + ec['versionsuffix'] | ||
|
||
def det_module_subdir(self, ec): | ||
|
||
release = 'releases' | ||
release_date = '' | ||
|
||
if ec['toolchain']['name'] == 'foss': | ||
release_date = ec['toolchain']['version'] | ||
elif ec['toolchain']['name'] == 'GCCcore': | ||
# please add a new GCCcore version | ||
# if you want to use a new toolchain version. | ||
if ec['toolchain']['version'] == '7.3.0': | ||
release_date = '2018b' | ||
elif ec['toolchain']['version'] == '6.3.0': | ||
release_date = '2017a' | ||
elif ec['toolchain']['version'] == '5.4.0': | ||
release_date = '2016b' | ||
elif ec['toolchain']['version'] == '4.9.3': | ||
release_date = '2016a' | ||
elif ec['toolchain']['version'] == '8.3.0': | ||
release_date = '2019b' | ||
elif ec['toolchain']['version'] == '10.2.0': | ||
release_date = '2020b' | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. As you mentioned, we can do a lot better here, although we would need to rely on easyconfig files (for the Perhaps it's not too terrible after all to hardcode this here, but then we need to maintain it when new common toolchain versions are defined, of course. If we do so, this should be improved so we have a constant on top that defines the relation between |
||
elif ec['toolchain']['name'] == 'GCC': | ||
# please add a new GCC version | ||
# if you want to use a new toolchain version. | ||
if ec['toolchain']['version'] == '7.3.0-2.30': | ||
release_date = '2018b' | ||
elif ec['toolchain']['version'] == '6.3.0-2.27': | ||
release_date = '2017a' | ||
elif ec['toolchain']['version'] == '5.4.0-2.26': | ||
release_date = '2016b' | ||
elif ec['toolchain']['version'] == '4.9.3-2.25': | ||
release_date = '2016a' | ||
elif ec['toolchain']['version'] == '8.3.0-2.32': | ||
release_date = '2019b' | ||
elif ec['toolchain']['version'] == '10.2.0': | ||
release_date = '2020b' | ||
elif ec['toolchain']['name'] == 'gompi': | ||
release_date = ec['toolchain']['version'] | ||
elif ec['toolchain']['name'] == 'fosscuda': | ||
release_date = ec['toolchain']['version'] | ||
elif ec['toolchain']['name'] in [DUMMY_TOOLCHAIN_NAME, SYSTEM_TOOLCHAIN_NAME]: | ||
release_date = '' | ||
release = 'General' | ||
else: | ||
release_date = "NOTFOUND" | ||
|
||
subdir = os.path.join(release, release_date) | ||
return subdir |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't believe that you can drop the toolchain/version suffix here without exposing yourself to lots of problems. How does this take into account that you can have, for example, OpenMPI with other compilers? Looking at a concrete example:
OpenMPI-4.0.3-GCC-9.3.0.eb
OpenMPI-4.0.3-gcccuda-2020a.eb
OpenMPI-4.0.3-iccifort-2020.1.217.eb
all of these map to the same module file but are clearly not the same, this lack of uniqueness is a big problem as EB will see them all as installed once the module file is created.
It seems that the MNS is by design focussed on
foss
alone which means it is heavily exposed to problems that may occur when mixing different toolchains (for example fortran module incompatability between Intel/GCC). Even atfoss
level the MNS relies on there being no shadowing of software with different toolchains (for example, Python withGCCcore
and the same version withGCC
), this is currently true in recent releases but there is no guarantee that sites themselves respect this.