Adding testing of a symlink tree install

[vsoch]

This is a first shot at discussion from #456 (comment).

See https://github.com/singularityhub/singularity-hpc/pull/502/files#diff-2fc3b9f958ccce7c952b2460ec359be8c54adb468a2e6908328c8537066d1419R365-R415 for usage.

Signed-off-by: vsoch [email protected]

[marcodelapierre]

So, as regards the symlink tree, this sounds like a great idea indeed!

To be honest, I was planning to generate this type of compact tree using a bash script on our HPC centre, so....certainly handy if SHPC can handle the creation of a more compact moduletree.

And I really like @muffato 's key idea of having SHPC generating a duplicate, symlinked tree, and warn users when overwriting occurs.

I haven't got the chance (and won't for a while) to have a look at the code edits of this PR, but here are Some thoughts:

• Feature name (silly philosophical thought): how about a more abstract one, such as module shortcuts (with its Windows 95 vibes ahah!), or compact modules?
• Use case for the feature: I would tend to think that admins/users will want to generate the compact module tree either for all modules, or for none at all, with no need for an "hybrid" setup (generate the compact tree only for "some" modules). @muffato what do you think? I am asking this because of the next point below..
• How to control the feature: similar to other features, I think it would be great if this could be controlled entirely via the settings yaml, without need for extra CLI flags. Eg as @vsoch said, having something like this in the settings yaml:

  # create an additional, compact module-tree, that only uses container <tool> and <version>
  # null to disable (default), otherwise specify install directory (eg $root_dir/module_shortcuts)
  module_shortcuts_base:    
  

  or, even, for better control:

  # create an additional, compact module-tree, that only uses container <tool> and <version>
  module_shortcuts:
    enabled: false
    # install directory
    base: $root_dir/module_shortcuts
  

• Option for applying this starting from namespace: I like this idea of yours, @vsoch! Maybe there can be additional settings entries to control whether this should be done or not? E.g.

    # use namespaces to create module shortcuts, instead of the generic <tool>/<version>
    use_namespace: false
  

• Possible pain point: ensure container module uninstallation does the right thing in the symlink tree
• Possible pain point: check possible spurious interactions with default_version for modules

that's it for now :)

[vsoch]

Oh those are some good points on the cleanup / uninstall - what we would do is generate the same symlink path and then remove it if it exists. if the user changes their symlink_home then it would be up to them to nuke / manage the whole thing. I'm only up briefly but here are some thoughts!

Feature name (silly philosophical thought): how about a more abstract one, such as module shortcuts (with its Windows 95 vibes ahah!), or compact modules?

I like those names but I'm worried it doesn't make it stupidly obvious what we've created. It's a "symlink tree" or a view of the main modules enabled by symlinks. A module shortcut or compact module I'm not sure I would know it's a symlink.

Use case for the feature: I would tend to think that admins/users will want to generate the compact module tree either for all modules, or for none at all, with no need for an "hybrid" setup (generate the compact tree only for "some" modules). @muffato what do you think? I am asking this because of the next point below..

I am thinking of it a bit like a view, and in which case I might actually want to manage the tree for my users, maybe not always providing all containers or tags in the view but some select set. So my thinking is we probably want to empower all use cases - being able to install a new moduble and say "add to the symlink tree" on the fly, but perhaps ALSO have a global flag in the settings to say "do this by default." I can add this tomorrow evening - I'll add:

# always generate an equivalent in the symlink tree on install
symlink_tree: true

to settings.yml and that will be equivalent to always providing the --symlink flag on installs. I also think we want some global commands to better manage things, e.g., given that we call the symlink tree a "view":

$ shpc view add <module>
$ shpc view remove <module>
$ shpc view add --all
$ shpc view remove --all

This could get very complicated very fast, e.g., allowing the user to create different or named views for different sets of modules in different places. My intuition is saying "keep it simple" and that we should allow one symlink home (and thus one view) for the user to control, otherwise the functionality will just be too confusing. Would a cluster really have use for managing more than one view?

How to control the feature: similar to other features, I think it would be great if this could be controlled entirely via the settings yaml, without need for extra CLI flags. Eg as @vsoch said, having something like this in the settings yaml:

haha yes I think this is what I just mentioned above, albeit with a different name! And I called the root symlink_home instead. I think instead of the nested structure my preference is to flatten things out, when possible, but I could be swayed to have a nested one.

use_namespace: false

So I'm not totally following here - right no we have the concept of namespaces, but that is to interact with modules for shpc. E.g., you can load a namespace ghcr.io/autamus and then just refer to anything under it with shpc, but that doesn't extend to the module software (e.g., you still load ghcr.io/autamus/. I think my concern here with adding custom namespaces is that if it's different for each module, we don't have a consistent way to find each one again on, for example, an uninstall (which i forgot to implement here).

Possible pain point: ensure container module uninstallation does the right thing in the symlink tree

omg yes I totally forgot to implement this!! I'll give it a go tomorrow. As long as the symlink tree home or root is provided, we should be able to predict the roots and do the uninstall akin to how we do in the regular module tree. If the user creates a tree and then changes it, they are on their own!

Possible pain point: check possible spurious interactions with default_version for modules

Can you give some examples here? I was noticing that the .version file didn't seem to be removing the "module" at the end, and I'm wondering if I created it incorrectly or there is something else we can do to better handle this default version.

Okay I'm going back to bed! I will do another round of work on this tomorrow evening after some more sleep and a workday 😆 There is also good discussion starting in #501 about updates, and I think we should spec out an update command group (and possibly deprecate the check functionality). E.g., it would be nice for the user to not to have to rely on pulling updates from github, which only happen once a month, and just to be able to run:

$ shpc update --all

but we need to distinguish between:

1. just updating the container.yaml files (digests and new tags akin to the github bot) but not doing any new installs
2. asking to install newest digests
3. asking to install newest tags OR digests
4. the same as 3 but also uninstalling the previous ones

Something to think about - how would y'all like the above interactions to look? Should there be different flags / options or a more interactive workflow?

Okay back to bed. Hooray Thursday coming up almost Friday!!

[vsoch]

okay updated with:

• the -c parameter to set / add/ remove any setting on the fly
• symlink_tree true/false in settings to say "always generate if symlink_home is defined no matter what"
• cleanup of symlink on uninstall

[marcodelapierre]

Agree on your feedback, in particular the "keep it simple" statement.
I would not overcomplicate these features unless there are explicit requests by SHPC users.

Two suggestions for consistency:

• if symlink_tree is the toggle in settings yaml, then I suggest having the CLI flag to be --symlink-tree
• I would suggest to use symlink_base for the path, for consistency with container_base and module_base

I think a "flat" settings structure, with tree and base/home separated is good.

And just to make sure I have understood it right, the symlink_tree itself has a structure based solely on container and , correct?

On my mention of namespaces, I was referring to this comment on yours in #456:

And I was thinking we could have a fourth open "install to same namespace" (with a "proceed at your own risk" ) sort of deal - if the user is installing a particular tag they could largely avoid the conflicts.

But I don't think it's a priority right now.

On the default_version, TLDR: refreshed my mind, I think it's all fine. Long story: when true, this feature generates a .version file in the directory. Yesterday I was thinking we need to check that this file is created/deleted appropriately in the symlink tree, but now I realise this should be fine, because the symlink points to the directory, so everything is already in the right place in the symlink tree.

Not diving in the update thread for now, not enough mental energies!

[vsoch]

okay!

1. --symlink-tree instead of --symlink
2. symlink_base instead of symlink_home

And just to make sure I have understood it right, the symlink_tree itself has a structure based solely on container and , correct?

It would just be based on the container name and version, so this means the user can technically install different modules (with the same base container name) to the same top level directory given they have the same versions. At first I was thinking we'd make them symlink the entire module directory (with version subfolders) but then I realized it was much easier to maintain one level lower, at the level of the version. So when we are parsing an install what we symlink is the module_dir folder.

And I was thinking we could have a fourth open "install to same namespace" (with a "proceed at your own risk" ) sort of deal - if the user is installing a particular tag they could largely avoid the conflicts.

So this is sort of the default now? Everything is "proceed at your own risk" and if a duplicate name/version is found the user ca choose to proceed or not. The other options in that list were based on implementing it one level up. I think this is simpler and probably more flexible to mix things up.

On the default_version, TLDR: refreshed my mind, I think it's all fine. Long story: when true, this feature generates a .version file in the directory. Yesterday I was thinking we need to check that this file is created/deleted appropriately in the symlink tree, but now I realise this should be fine, because the symlink points to the directory, so everything is already in the right place in the symlink tree.

You actually have a good point - I think the .version file is generated one level above that, so we would need to clean it up. I'm not doing that at the moment in this PR but I can take a look to add it.

Not diving in the update thread for now, not enough mental energies!

TOTALLY agree lets focus on the symlink tree and the added config variable here - nothing else exists until we got this underway and merged!

[marcodelapierre]

makes sense!

so, on default_version, indeed, if what is symlinked is "module_dir" (which is version specific), then the symlink feature needs to take care of creating/deleting the .version file in the symlink tree.

[vsoch]

so, on default_version, indeed, if what is symlinked is "module_dir" (which is version specific), then the symlink feature needs to take care of creating/deleting the .version file in the symlink tree.

Yes exactly! So I do that same check on creation of the symlink directory and create the .version file if needed:

# If we don't have a version file in root, create it
if self.module_extension != "tcl" and self.settings.default_version == True:
    version_file = os.path.join(os.path.dirname(symlink_path), ".version")
    if not os.path.exists(version_file):
        Path(version_file).touch()

and just this evening I added the same check on cleanup - look to see if the parent of the version folder (which is the one symlinked) is empty or JUST has the version file, and if so, delete it (it's empty)

# If the parent of the symlink only has zero files OR one file .version, cleanup
files = os.listdir(parent_dir)
if len(files) == 0 or (len(files) == 1 and files[0] == ".version"):
    shutil.rmtree(parent_dir)

[muffato]

Hi. Thanks a lot for writing all of this.
The first bug I'm facing is that the global setting symlink_tree doesn't seem to work. The only way I can get the symlinks is by adding the --symlink-tree on the command-line. Could it be because it's spelt with an hyphen and an underscore throughout the codebase ?

$ git grep symlink_tree
docs/getting_started/user-guide.rst:   * - symlink_tree
shpc/main/modules/__init__.py:        symlink = self.settings.symlink_tree is True or symlink
shpc/main/schemas.py:    "symlink_tree": {"type": "boolean"},
shpc/settings.yml:symlink_tree: true
$ git grep symlink-tree
.github/workflows/test.yml:          shpc install python:3.9.5-alpine --symlink-tree
docs/getting_started/user-guide.rst:     - If set, where you want to install a simplified module tree to using ``--symlink-tree``
docs/getting_started/user-guide.rst:     - If set to true, ALWAYS generate a symlink tree given that a symlink base is defined regardless of ``--symlink-tree`` f
docs/getting_started/user-guide.rst:    $ shpc install ghcr.io/autamus/clingo --symlink-tree
docs/getting_started/user-guide.rst:    $ shpc install ghcr.io/autamus/samtools --symlink-tree
docs/getting_started/user-guide.rst:        shpc install $module --symlink-tree
shpc/client/__init__.py:        "--symlink-tree",

[marcodelapierre]

So, if I understand correctly, @muffato , your changes impact the dir/file naming for the symlink tree, from <tool>/<version>/module.ext to <tool>/<version> .

This is a great point, which however I think requires some extra care.

In fact, this change is not only cosmetic, but it also has functional implications, that are not documented in the Lmod/EnvModules but still existing. The first syntax (one module.ext file per tool version) provides a way to disable default versions for modules, which is something some HPC centres may want to enforce (we're one of them ;) ). So, switching from syntax one to syntax two effectively forces the activation of default module versions.
By "default versions", I mean that with the 2nd syntax, module load <tool> is legal (the module system will pick a version), whereas with the 1st syntax, users are forced to go with module load <tool>/<version>.

We had similar conversations in this Github project in the past, and @vsoch ended up implementing a setting default_version to control this functionality.
Therefore, I would strongly suggest to edit your implementation, so that it is controlled by the default_version setting in the yaml file: enable the new naming convention for the symlink tree only if such setting is true.

I hope this makes sense, if not I am happy to chat further.

[muffato]

Hi @marcodelapierre

That doesn't seem to be the case on our installation of modules here (Modules Release 4.3.0 (2019-07-26), on TCL)

$ module avail
------------------------------------------------------------------------- /software/treeoflife/shpc/modules -------------------------------------------------------------------------
ghcr.io/autamus/bedtools2/2.30.0/module.tcl                                                    golang/1.18-rc/module.tcl                                     
ghcr.io/autamus/bwa/0.7.17/module.tcl                                                          quay.io/biocontainers/bwa-mem2/2.2.1--h9a82719_1/module.tcl   
ghcr.io/autamus/iq-tree/2.1.3/module.tcl                                                       quay.io/biocontainers/bwa/0.7.17--h84994c4_4/module.tcl       
ghcr.io/autamus/picard/2.26.5/module.tcl                                                       quay.io/biocontainers/cooler/0.8.6--py_0/module.tcl           
ghcr.io/autamus/rust/1.54.0/module.tcl                                                         quay.io/biocontainers/cooler/0.8.11--pyh3252c3a_0/module.tcl  
gitlab-registry.internal.sanger.ac.uk/sanger-pathogens/docker-images/fasta/36.3.8h/module.tcl  quay.io/biocontainers/samtools/1.14--hb421002_0/module.tcl    

$ type go
bash: type: go: not found

$ module load golang

$ module list
Currently Loaded Modulefiles:
 1) golang/1.18-rc/module.tcl

$ type go
go is a function

It's able to load a module from a prefix, as long as the prefix is unique (it wouldn't be able to load ghcr.io).

What version do you have installed ?

Regardless, I understand how useful preventing a default version to be loaded, though, and I'd be happy to keep that possibility. We use this trick here in a different module tree, that I'd be looking into putting in our shpc deployment at some point: create a .version for every software, with this content:

#%Module
set ModulesVersion "please_specify_a_version_number"

[marcodelapierre]

cool! - didn't have the chance to test, but feel free to go ahead for what I am concerned, sounds like a great step forward

[marcodelapierre]

Hi @vsoch, @muffato,
so I think I have just found a good way of enforcing default_version = false for Lua, both with and without symlinks.
It's an edit in the Jinja template, so the price to pay is that we need to pass to it the logical of default_version, so that a decision can be made.
This is the snippet:

if (mode() == "load") then
  if ( myModuleUsrName() ~= myModuleFullName() and myModuleUsrName() ~= string.gsub(myModuleFullName(),"/module$","") ) then
    LmodError("You must specify module <name>/<version>.")
  end
end

Basically, it causes an error if the module name to be loaded does not match one of <tool>/<ver>/module or <tool>/<ver>, hence rejecting .
I based it off the implementation of the Lmod function requireFullName() (see https://lmod.readthedocs.io/en/latest/050_lua_modulefiles.html), which unfortunately is only available in newer Lmod versions.
The error message is just a suggestion.

So the proposed new implementation of deault_version for Lua is as follows:

• if false, enable the snippet above in the Jinja template
• this will work for both the full tree and the symlinked tree
• eventually, it is now possible to always create the .version file, even with false (it would be empty), because the logic above will prevail anyway

My understanding is that in this current PR, symlink_tree is not possible with lua and default_version=false; with this change, they would become possible.

I am not able to implement it right now, but I think it is quite a small edit to do, the main thing to keep in mind is passing default_version to the Jinja template rendering.

What do you think?
Thanks,

[vsoch]

I'll defer to @muffato, who (I think) has a setup to test these out! Note we are working on a double nested PR - there is a PR to this PR from @muffato that I've PR'd to, muffato#1.

@muffato if you have preference for an implementation let me know and I'll update there, and we can eventually work our way back here! And indeed we have a conflict since a merge this morning - don't worry about this for now, since I've worked on both I'll be happy to do the resolution when the time comes

[vsoch]

TODO: when this set of PRs is merged remember to ping @dtrudg about giving a talk for SHPC at a singularity community call.

[muffato]

Finally getting back to @marcodelapierre 's comment.
To clarify, symlink_tree is possible with lua and default_version=false, but the symlinks have to be named clingo/5.5.1/module. Your solution should work and allow clingo/5.5.1, I can try it, but you said "is only available in newer Lmod versions". How limiting would that be for shpc ?

[marcodelapierre]

Hi @muffato,

apologies I wast not clear in my message.
The solution from my snippet above works with older Lmod versions, too.

The solution that would NOT work with older versions is that which uses the new Lmod function requireFullName() (which I found in the latest Lmod docs).

To make my solution above work with older versions, I have not used such new function directly. Instead, I have explicitly used the implementation of such function. And, just to be clear, It's this one, copy-pasted from above:

if (mode() == "load") then
  if ( myModuleUsrName() ~= myModuleFullName() and myModuleUsrName() ~= string.gsub(myModuleFullName(),"/module$","") ) then
    LmodError("You must specify module <name>/<version>.")
  end
end

I don't know exactly when these my.. functions were introduced in Lmod, but I can tell you it works at least with Lmod 7.6.1, which was released in 2017.

Does it make more sense Matthieu?

[muffato]

Hiya. 2 comments:
As things stand there is a bug with Lmod:

singularity-hpc/shpc/main/modules/__init__.py

Line 191 in 18a76ee

if self.module_extension == "lua" and self.settings.default_version == True:

and

singularity-hpc/shpc/main/modules/__init__.py

Line 210 in 18a76ee

if self.module_extension == "lua" and self.settings.default_version == False:

only support the previously accepted values True and False, not their new synonyms module_sys and null.
Easy to fix but I've also given a try to @marcodelapierre's trick and I can confirm it works. It would remove the need for checking the value of default_version in these two lines, and do it in the template instead.

Just need to make a couple more tests before committing and pushing. One thing I want to raise first is that it may look confusing for users:

$ module avail
   bwa-mem2/2.2.1--hd03093a_2    bwa/0.7.17--h7132678_9 (D)    golang/1.18-rc               samtools/1.15--h3843a85_0 (D)
   bwa/0.7.17--h5bf99c6_8        clingo/5.5.1                  samtools/1.14--hb421002_0

(only setting: default_version set to null)
Notice how Lmod apparently assigns a default version. But then:

$ module load bwa
Lmod has detected the following error:  You must specify module <name>/<version>. 
While processing the following module(s):
    Module fullname         Module Filename
    ---------------         ---------------
    bwa/0.7.17--h7132678_9  /nfs/users/nfs_m/mm49/nfs/scratch123/shpc/singularity-hpc/symlinks/bwa/0.7.17--h7132678_9.lua

How does that sound ?

[vsoch]

Sounds good! Just to give you a heads up on this PR - the rebase is gnarly so I'll need to do it locally, probably after dinner tonight!

[vsoch]

Huzzah tests pass! Ok I'm exhausted tonight and need to work on some other things, but let's all give this another look (if not the weekend next week) and have more discussion. Thanks for all the hard work y'all this is looking good!

[muffato]

Cool 🚀 , thanks ! I'll push my last updates to my branch later today, and make a PR against your branch ? (not everything could fit within suggestions on this PR).

[vsoch]

Yes that would be great!

[marcodelapierre]

@muffato you're right, that spurious "D" for Default is inconsistent with the failing module without Default.
At this stage I don't have a solution though.

[muffato]

@marcodelapierre . I still pushed the change to #539 . Maybe the error message should have something like "Default modules are disabled by your systems administrator. Please specify a version" to make it clearer it's not Lmod's fault ?
I consider the branch in #539 to fully complete the implementation of the symlink tree – according to my expectations ;).

[marcodelapierre]

Good point @muffato, I like the alternative message you propose!

[muffato]

Done

[vsoch]

See extension of this work into "shpc views" here: #545

[vsoch]

Preceded by #545