Directory structure

[bedroge]

We already have an open issue to discuss the naming scheme for the repositories (see #4), but inside the prod/test repositories we need to come up with / decide on a directory structure.

This is the concept we showed during the EESSI & CVMFS meeting:

.
└── cvmfs/
    ├── cvmfs-config.eessi-hpc.org/
    │   └── etc/
    │       └── cvmfs/
    ├── test.eessi-hpc.org
    └── prod.eessi-hpc.org/
        └── 2020.06/
            ├── compat/
            │   ├── aarch64
            │   ├── ppc64le
            │   └── x86_64/
            │       ├── bin
            │       ├── etc
            │       ├── lib64
            │       └── usr
            └── software/
                ├── aarch64
                ├── ppc64le
                └── x86_64/
                    ├── amd/
                    │   └── zen2
                    └── intel/
                        ├── haswell
                        └── skylake/
                            ├── modules
                            └── software/
                                ├── GROMACS/
                                │   ├── 2019.3-fosscuda-2019b
                                │   └── 2020.1-foss-2020a-Python-3.8.2
                                └── TensorFlow/
                                    └── 2.2.0-fosscuda-2019b-Python-3.7.4

(edit it here)

With variant symlinks we can point a fixed directory to the right tree, based on the client's architecture.

[boegel]

Oh that variant symlinks trick is neat, it can help us quite a bit to shorter the path to something like /cvmfs/prod.eessi-hpc.org/{modules,software} (which would then point to /cvmfs/prod.eessi-hpc.org/2020.06/software/x86_64/intel/haswell/{modules,software} for example...

[bedroge]

After the discussions that we recently had in the monthly meeting and the brainstorm about the 2021.12 pilot, I was rethinking our directory structure. In my opinion it would be nice to move all the actual versions to their own directory (e.g. versions), and use (variant? maybe that doesn't really make sense here?) symlinks in the root of the repo to point to specific versions.

I also still like the idea of having variant symlinks modules and software, as was already discussed above and also recently proposed by @ocaisa in EESSI/software-layer#146. We could put those either in the root of the repo, in which case they also have to include a version, and/or we could put them in the directories of the versions. In the latter case, we have to rename our current software directory, though, and you still don't have a completely invariant path.

So, using the first approach with the version part of the variant symlink, it would look like:

.
└── cvmfs/
    ├── test.eessi-hpc.org
    └── prod.eessi-hpc.org/
        ├── README
        ├── modules -> '$(EESSI_ARCH_DIR)/software'
        ├── software -> '$(EESSI_ARCH_DIR)/software'
        ├── latest -> '$(EESSI_LATEST:-versions/2022.03)
        ├── oldstable '$(EESSI_OLDSTABLE:-versions/2021.06)
        ├── stable -> '$(EESSI_STABLE:-versions/2021.12)
        └── versions/
            ├── 2021.06
            ├── 2021.12
            └── 2022.03/
                ├── compat/
                │   ├── aarch64
                │   ├── ppc64le
                │   └── x86_64/
                │       ├── bin
                │       ├── etc
                │       ├── lib64
                │       └── usr
                └── software/
                    ├── aarch64
                    ├── ppc64le
                    └── x86_64/
                        ├── amd/
                        │   └── zen2
                        └── intel/
                            ├── haswell
                            └── skylake/
                                ├── modules
                                └── software/
                                    ├── GROMACS/
                                    │   ├── 2019.3-fosscuda-2019b
                                    │   └── 2020.1-foss-2020a-Python-3.8.2
                                    └── TensorFlow/
                                        └── 2.2.0-fosscuda-2019b-Python-3.7.4

(source

One could then set EESSI_ARCH_DIR to something like versions/2021.12/software/intel/haswell or latest/software/intel/haswell in the local CVMFS configuration file.

What do you think? Does this make sense? If so, it would be nice to change this for the next pilot already (except that we can't move 2021.06, but that's a minor and temporary detail...).

[ocaisa]

I think that rather than give control to the user over what latest means we should instead put in place a site_production variable symlink that we ignore but they can use to control the update cycle. That will allow us to write reliable scripts.

Here it would seem that latest is like our devel branch, is my understanding correct? Our current use of latest looks more like stable

[ocaisa]

So I would like to to see variable symlinks that they control for modules, software and site_production

[bedroge]

I think I like the idea of having one variant symlink site_production instead of allowing sites to override what define as stable/latest. And would we also point the software and modules symlink to that site_production version? So something like this?

.
└── cvmfs/
    ├── test.eessi-hpc.org
    └── prod.eessi-hpc.org/
        ├── README
        ├── modules -> 'site_production/$(EESSI_SITE_ARCH)/modules'
        ├── software -> 'site_production/$(EESSI_SITE_ARCH)/software'
        ├── latest -> versions/2022.03
        ├── oldstable -> versions/2021.06
        ├── stable -> versions/2021.12
        ├── site_production -> 'versions/$(EESSI_SITE_PRODUCTION_VERSION)'
        └── versions/
            ├── 2021.06
            ├── 2021.12
            └── 2022.03

(full listing here )

Where EESSI_SITE_ARCH would be, for instance, x86_64/intel/haswell, and EESSI_SITE_PRODUCTION_VERSION=2021.12 or EESSI_SITE_PRODUCTION_VERSION=stable.

@ocaisa I think dev/tests should in the end go into a separate repo, but latest in prod could be a new installation that should be ready to use. In the meeting about the next pilot we talked about having a sort of LTS release, so in this case stable would point to that LTS version, while latest is a newer non-LTS version (maybe stable could also be named lts, but that's a detail).

[ocaisa]

I'm pretty unclear about what stable and latest imply. Is this related to the compatibility layer only? Could it happen that the same toolchains appear in both? I think I would like to be able to mix toolchains from different compat layers (in a module hierarchy), there's no reason I can't do this so in my mind the choice of compat layer was a moving target based on the toolchain I want to contribute to

[ocaisa]

I have another proposal here based on the idea of being able to use architecture-portable workflows:

.
└── cvmfs/
    ├── test.eessi-hpc.org
    └── prod.eessi-hpc.org/
        ├── README
        ├── latest -> versions/2022.03
        ├── oldstable -> versions/2021.06
        ├── stable -> versions/2021.12
        ├── site_production -> 'versions/$(EESSI_SITE_PRODUCTION_VERSION)'
        └── versions/
            ├── 2021.06
            ├── 2021.12
            └── 2022.03/
                ├── portable_modules -> 'software/$(EESSI_SITE_ARCH)/modules'
                ├── portable_software -> 'software/$(EESSI_SITE_ARCH)/software'
                ├── compat/
                │   ├── aarch64
                │   ├── ppc64le
                │   └── x86_64/
                │       ├── bin
                │       ├── etc
                │       ├── lib64
                │       └── usr
                └── software/
                    ├── aarch64
                    ├── ppc64le
                    └── x86_64/
                        ├── amd/
                        │   └── zen2
                        └── intel/
                            ├── haswell
                            └── skylake/
                                ├── modules
                                └── software/
                                    ├── GROMACS/
                                    │   ├── 2019.3-fosscuda-2019b
                                    │   └── 2020.1-foss-2020a-Python-3.8.2
                                    └── TensorFlow/
                                        └── 2.2.0-fosscuda-2019b-Python-3.7.4

(source)
This way, if I make the assumption that the admin has set the ARCH setting whereever my workflow lands, I can use full paths to executables that are arch-independent while still being arch-optimised (for example, .../versions/2022.03/portable_software/GROMACS/2020.1-foss-2020a-Python-3.8.2/bin/gmx). For something like TensorFlow, I would have to use modules since I need a full environment but that can still be done with module use .../versions/2022.03/portable_modules.

One can still use site_production/portable_modules and site_production/portable_software to give similar behaviour to the previous proposal.

[ocaisa]

One could even consider not making portable_modules a symlink at all, but instead constructing a new module tree(s) based on what is found in portable_software...I wonder if that would work? That would be much better when using this approach in a queueing system on heterogeneous systems, since the path inside the modules would be the arch-independent path and that would find the right exectuables on the compute nodes.