Update spack-stack-dev to spack v0.23 #487
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The already concrete specs in an environment are now among the reusable specs for the concretizer.
This includes concrete specs from all include_concrete environments.
In addition to this change to the default reuse, `environment` is added as a reuse type for
the concretizer config. This allows users to specify:
spack:
concretizer:
# Reuse from this environment (including included concrete) but not elsewhere
reuse:
from:
- type: environment
# or reuse from only my_env included environment
reuse:
from:
- type:
environment: my_env
# or reuse from everywhere
reuse: true
If reuse is specified from a specific environment, only specs from that environment will be reused.
If the reused environment is not specified via include_concrete, the concrete specs will be retried
at concretization time to be reused.
Signed-off-by: Ryan Krattiger <[email protected]>
Co-authored-by: Gregory Becker <[email protected]>
Fixes an issue reported where `spack env depfile` + `make -j` would
non-deterministically refuse to mark all environment roots explicit.
`update_explicit` had the pattern
```python
rec = self._data[key]
with self.write_transaction():
rec.explicit = explicit
```
but `write_transaction` may reinitialize `self._data`, meaning that
mutating `rec` won't mutate `self._data`, and the changes won't be
persisted.
Instead, use `mark` which has a correct implementation.
Also avoids the essentially incorrect early return in `update_explicit`
which is a pattern I don't think belongs in database.py: it branches on
possibly stale data to realize there is nothing to change, but in reality
it requires a write transaction to know that for a fact, but that would
defeat the purpose. So, leave this optimization to the call site.
* Introduce support for ArmPL and ACfL 24.10 This patch introduces the possibility of installing armpl-gcc and acfl 24.10 through spack. It also addressed one issue observed after PR spack#46594 * Fix Github action issues. - Remove default URL - Reinstate default OS for ACfL to RHEL.
Co-authored-by: Scot Halverson <[email protected]>
- Merging sycl2020usm and sycl2020acc into sycl2020 and the submodel=acc/usm variant is introduced - implementation is renamed to option - impl ( fortran implementation options) renamed to foption - sycl_compiler_implementation and thrust_backend - stddata,stdindices,stdranges to a single std with std_submodel introduction - std_use_tbb to be boolean; also changed model filtering algorithm to make sure that it only picks model names - Modified comments to clear confusion with cuda_arch cc_ and sm_ prefix appends - Deleted duplicate of cuda_arch definition from +omp - CMAKE_CXX_COMPILER moved to be shared arg between all models except tbb and thrust - Replaced sys.exit with InstallError and created a dictionary to simplify things and eliminate excess code lines doing same checks - Replaced the -mcpu flags to -march since it is deprecated now - Replaced platform.machine with spec.target - Removing raja_backend, introducing openmp_flag,removing -march flags,clearing debugging print(), removing excess if ___ in self.spec.variants - [FIX] Issue where Thrust couldn't find correct compiler (it requires nvcc) - [FIX] Fortran unsupported check to match the full string - [FIX] RAJA cuda_arch to be with sm_ not cc_ - dir= option is no longer needed for kokkos - dir is no longer needed - [omp] Adding clang support for nvidia offload - SYCL2020 offload to nvidia GPU - changing model dependency to be languages rather than build system - removing hardcoded arch flags and replacing with archspec - removing cpu_arch from acc model --------- Signed-off-by: Todd Gamblin <[email protected]> Co-authored-by: Greg Becker <[email protected]> Co-authored-by: Kaan Olgu <[email protected]> Co-authored-by: Todd Gamblin <[email protected]>
) Co-authored-by: Bernhard Kaindl <[email protected]>
…pack#45592) Bumps [docutils](https://docutils.sourceforge.io) from 0.20.1 to 0.21.2. --- updated-dependencies: - dependency-name: docutils dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Bernhard Kaindl <[email protected]>
* geant4: make downloading data dependency optional This PR makes downloading the data repository of the Geant4 spec optional by adding a sticky, default-enabled variant which controls the dependency on `geant4-data`. This should not change the default behaviour, but should allow users to choose whether or not they want the data directory. * Add comment * Update env variable * Generic docs * Buildable false
* cp2k: protect against newer libxc * Compat bound for libxc
* Add openfast version 3.5.4 * remove commits
This changes `Spec` serialization to include information about propagation for abstract specs. This was previously not included in the JSON representation for abstract specs, and couldn't be stored. Now, there is a separate `propagate` dictionary alongside the `parameters` dictionary. This isn't beautiful, but when we bump the spec version for Spack `v0.24`, we can clean up this and other aspects of the schema.
* don't concretize in CI if changed packages are not in stacks Signed-off-by: Todd Gamblin <[email protected]> * Generate noop job when no specs to rebuild due to untouched pruning * Add test to verify skipping generate creates a noop job * Changed debug for early exit --------- Signed-off-by: Todd Gamblin <[email protected]> Co-authored-by: Todd Gamblin <[email protected]>
Currently, the `concretizer:unify:` config option only affects environments. With this PR, it now affects any group of specs given to a command using the `parse_specs(*, concretize=True)` interface. - [x] implementation in `parse_specs` - [x] tests - [x] ensure all commands that accept multiple specs and concretize use `parse_specs` interface --------- Co-authored-by: Todd Gamblin <[email protected]> Signed-off-by: Todd Gamblin <[email protected]>
Bumps [python-levenshtein](https://github.com/rapidfuzz/python-Levenshtein) from 0.26.0 to 0.26.1. - [Release notes](https://github.com/rapidfuzz/python-Levenshtein/releases) - [Changelog](https://github.com/rapidfuzz/python-Levenshtein/blob/main/HISTORY.md) - [Commits](rapidfuzz/python-Levenshtein@v0.26.0...v0.26.1) --- updated-dependencies: - dependency-name: python-levenshtein dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Currently, the schema reads:
from:
- type:
environment: path_or_name
but this can't be extended easily to other types, e.g. to buildcaches,
without duplicating the extension keys. Use instead:
from:
- type: environment
path: path_or_name
According to root-project/root#7160, if `-Dcocoa=ON` build must also be configured with `-Dopengl=ON`, since otherwise the build encounters missing includes. This is/was a silent failure in ROOT CMake, but I believe has been made an explicit failure some time this year.
* sherpa: add v3.0.1 * sherpa: no depends_on py-setuptools
…ages. (spack#46729) This PR provides complementary 2 features: 1. An augmentation to the package language to express ABI compatibility relationships among packages. 2. An extension to the concretizer that can synthesize splices between ABI compatible packages. 1. The `can_splice` directive and ABI compatibility We augment the package language with a single directive: `can_splice`. Here is an example of a package `Foo` exercising the `can_splice` directive: class Foo(Package): version("1.0") version("1.1") variant("compat", default=True) variant("json", default=False) variant("pic", default=False) can_splice("[email protected]", when="@1.1") can_splice("[email protected]", when="@1.0+compat") can_splice("[email protected]+compat", when="@1.0+compat", match_variants="*") can_splice("[email protected]", [email protected]~compat", match_variants="json") Explanations of the uses of each directive: - `can_splice("[email protected]", when="@1.1")`: If `[email protected]` is the dependency of an already installed spec and `[email protected]` could be a valid dependency for the parent spec, then `[email protected]` can be spliced in for `[email protected]` in the parent spec. - `can_splice("[email protected]", when="@1.0+compat")`: If `[email protected]` is the dependency of an already installed spec and `[email protected]+compat` could be a valid dependency for the parent spec, then `[email protected]+compat` can be spliced in for `[email protected]+compat` in the parent spec - `can_splice("[email protected]", when="@1.0+compat", match_variants="*")`: If `[email protected]+compat` is the dependency of an already installed spec and `[email protected]+compat` could be a valid dependency for the parent spec, then `[email protected]+compat` can be spliced in for `[email protected]+compat` in the parent spec, provided that they have the same value for all other variants (regardless of what those values are). - `can_splice("[email protected]", [email protected]~compat", match_variants="json")`:If `[email protected]` is the dependency of an already installed spec and `[email protected]~compat` could be a valid dependency for the parent spec, then `[email protected]~compat` can be spliced in for `[email protected]` in the parent spec, provided that they have the same value for their `json` variant. 2. Augmenting the solver to synthesize splices ### Changes to the hash encoding in `asp.py` Previously, when including concrete specs in the solve, they would have the following form: installed_hash("foo", "xxxyyy") imposed_constraint("xxxyyy", "foo", "attr1", ...) imposed_constraint("xxxyyy", "foo", "attr2", ...) % etc. Concrete specs now have the following form: installed_hash("foo", "xxxyyy") hash_attr("xxxyyy", "foo", "attr1", ...) hash_attr("xxxyyy", "foo", "attr2", ...) This transformation allows us to control which constraints are imposed when we select a hash, to facilitate the splicing of dependencies. 2.1 Compiling `can_splice` directives in `asp.py` Consider the concrete spec: [email protected]%[email protected] arch=linux-ubuntu22.04-icelake build_system=autotools ^bar ... It will emit the following facts for reuse (below is a subset) installed_hash("foo", "xxxyyy") hash_attr("xxxyyy", "hash", "foo", "xxxyyy") hash_attr("xxxyyy", "version", "foo", "2.72") hash_attr("xxxyyy", "node_os", "ubuntu22.04") hash_attr("xxxyyy", "hash", "bar", "zzzqqq") hash_attr("xxxyyy", "depends_on", "foo", "bar", "link") Rules that derive abi_splice_conditions_hold will be generated from use of the `can_splice` directive. They will have the following form: can_splice("[email protected]+a", when="@1.0.1+a", match_variants=["b"]) ---> abi_splice_conditions_hold(0, node(SID, "foo"), "foo", BaseHash) :- installed_hash("foo", BaseHash), attr("node", node(SID, SpliceName)), attr("node_version_satisfies", node(SID, "foo"), "1.0.1"), hash_attr("hash", "node_version_satisfies", "foo", "1.0.1"), attr("variant_value", node(SID, "foo"), "a", "True"), hash_attr("hash", "variant_value", "foo", "a", "True"), attr("variant_value", node(SID, "foo"), "b", VariVar0), hash_attr("hash", "variant_value", "foo", "b", VariVar0). 2.2 Synthesizing splices in `concretize.lp` and `splices.lp` The ASP solver generates "splice_at_hash" attrs to indicate that a particular node has a splice in one of its immediate dependencies. Splices can be introduced in the dependencies of concrete specs when `splices.lp` is conditionally loaded (based on the config option `concretizer:splice:True`. 2.3 Constructing spliced specs in `asp.py` The method `SpecBuilder._resolve_splices` implements a top-down memoized implementation of hybrid splicing. This is an optimization over the more general `Spec.splice`, since the solver gives a global view of exactly which specs can be shared, to ensure the minimal number of splicing operations. Misc changes to facilitate configuration and benchmarking - Added the method `Solver.solve_with_stats` to expose timers from the public interface for easier benchmarking - Added the boolean config option `concretizer:splice` to conditionally load splicing behavior Co-authored-by: Greg Becker <[email protected]>
…pack#47567) benchmarked to test if that is a correct decision.
`spack spec` output has looked like this for a while: ```console > spack spec /v5fn6xo /wd2p2v7 Input spec -------------------------------- - /v5fn6xo Concretized -------------------------------- [+] [email protected]%[email protected]~docs+shared build_system=generic certs=mozilla arch=darwin-sequoia-m1 [+] ^ca-certificates-mozilla@2023-05-30%[email protected] build_system=generic arch=darwin-sequoia-m1 ... Input spec -------------------------------- - /wd2p2v7 Concretized -------------------------------- [+] [email protected]%[email protected] build_system=python_pip arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=generic arch=darwin-sequoia-m1 ``` But the input spec is right there on the CLI, and it doesn't add anything to the output. Also, since spack#44843, specs concretized in the CLI line can be unified, so it makes sense to display them as we did in spack#44489 -- as one multi-root tree instead of as multiple single-root trees. With this PR, concretize output now looks like this: ```console > spack spec /v5fn6xo /wd2p2v7 [+] [email protected]%[email protected]~docs+shared build_system=generic certs=mozilla arch=darwin-sequoia-m1 [+] ^ca-certificates-mozilla@2023-05-30%[email protected] build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]~guile build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]+cpanm+opcode+open+shared+threads build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]~debug~pic+shared build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools libs=shared,static arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools patches=bbf97f1 arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]~symlinks+termlib abi=none build_system=autotools patches=7a351bc arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]+compat+new_strategies+opt+pic+shared build_system=autotools arch=darwin-sequoia-m1 [+] ^gnuconfig@2022-09-17%[email protected] build_system=generic arch=darwin-sequoia-m1 [+] [email protected]%[email protected] build_system=python_pip arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=generic arch=darwin-sequoia-m1 [-] ^[email protected]%[email protected] build_system=generic arch=darwin-sequoia-m1 ... ``` With no input spec displayed -- just the concretization output shown as one consolidated tree and multiple roots. - [x] remove "Input Spec" section and "Concretized" header from `spack spec` output - [x] print concretized specs as one BFS tree instead of multiple --------- Signed-off-by: Todd Gamblin <[email protected]> Co-authored-by: Harmen Stoppels <[email protected]>
Automatic splicing say `Spec` grow a `__len__` method but it's only used
in one place and it's not clear the semantics are useful elsewhere. It also
runs the risk of Specs one day being confused for other types of containers.
Rather than introduce a new function for one algorithm, let's use a more
specific method in the splice code.
- [x] Use topological ordering in `_resolve_automatic_splices` instead of
sorting by node count
- [x] delete `Spec.__len__()` and `Spec.__bool__()`
---------
Signed-off-by: Todd Gamblin <[email protected]>
Co-authored-by: Greg Becker <[email protected]>
Co-authored-by: Massimiliano Culpo <[email protected]>
5 tasks
…remove unused variable
… feature/update_to_spack_v0p23
climbfuji
force-pushed
the
feature/update_to_spack_v0p23
branch
from
db9efb7 to
980763c
Compare
climbfuji
changed the title
climbfuji
requested review from
stiggy87,
srherbener,
RatkoVasic-NOAA and
AlexanderRichert-NOAA
srherbener
approved these changes
Dec 5, 2024
|
I'm going to do a test install based on JCSDA/spack-stack#1391 |
AlexanderRichert-NOAA
approved these changes
Dec 5, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Pull in spack release v0.23. Only merge conflict to check is for
py-cryptography.Two bug fixes are applied here and sent upstream to spack develop:
See JCSDA/spack-stack#1391 for more information