Skip to content

VSC easyconfigs fork discussion (2023 02 16)

Jump to bottom
Kenneth Hoste edited this page Feb 16, 2023 · 1 revision

VSC easyconfigs fork discussion (2023-02-16)

(Alex/Kenneth)

Situation

  • The general workflow across all sites is that individual installers keep their own fork (or folder) of easyconfigs
  • New installations are logged in vsc-software-stack arbitrarily at the discretion of each installer
  • New developments are pushed upstream arbitrarily at the discretion of each installer

Goals

  • Single library of easyconfigs for all software installed in VSC clusters
  • Place to store WIP easyconfigs (replace folders in vsc-software-stack)
  • Not a place for private easyconfigs (licensed software, secrets)
  • Public GitHub repository

Proposal

Option A: Single filetree

.
└── easyconfigs
  • Fork of upstream easyconfigs
  • Relaxed version restrictions
  • Addtions through PRs with self-test reports

Pros:

  • Highest chance to avoid duplication of work thanks to file collisons
  • Forces sites to converge on common easyconfigs

Cons:

  • Keeping the fork in-sync with upstream will need care due to conflicts
  • Difficult to also gather WIP easyconfigs with a single filetree
  • Forces sites to converge on common easyconfigs

Option B: Three filetrees

.
├── easybuilders/easyconfigs
├── vsc
└── wip
  • Mirror of upstream easyconfigs kept in clean state synced to develop branch (i.e. git submodule)
  • Fork of upstream easyconfigs with extra folders in vsc branch
  • Folder vsc holds filetree of easyconfigs installed in VSC clusters that are not in develop
  • Folder wip holds what is being worked-on (i.e. current folders in vsc-sofwtare-stack)

Pros:

  • Everything is still hold in a single place
  • Avoids maintenance cost of having to keep a fork in sync with changes upstream and deal with merge conflicts (not relevant if we use a proper fork)
  • Allows to have 2 paths to contribute:
    • vsc only accepts through PR with test
    • wip is free push
    • separate protected branch for vsc that requires going through PRs?
    • also require use of PRs for wip subdir, but don't require test report there?
  • Encourages sites to converge on common easyconfigs

Cons:

  • Unclear how to get CI tests working in this setup
  • Forces sites to converge on common easyconfigs

Option C: Per-site filetrees

.
├── @easybuilders
├── @antwerp
├── @brussel
├── @leuven
├── @gent
└── wip
  • Mirror of upstream easyconfigs kept in clean state synced to develop branch (i.e. git submodule)
  • Mirror of per-site repos with their easyconfigs (i.e. git submodule)
  • Folder wip holds what is being worked-on (i.e. current folders in vsc-sofwtare-stack)

Pros:

  • Everything is still hold in a single place
  • Avoids maintenance cost of having to keep a fork in sync with changes upstream and deal with merge conflicts
  • Lowest barrier of entry: no PRs as each site manages their own repo in their own way and wip is free push

Cons:

  • Lacks community/network dynamic: every site can continue to work on its own corner (status quo will not change)
  • Different easyconfigs for same software version depending on site

Notes

  • structure

    • Option A

      • too problematic for a shared fork across VSC sites (merge conflicts, etc.)

    • Option B

      • looks good, but using proper fork instead of git submodule
      • use site-specific free-to-push folders (site/{brussels, ghent, leuven, antwerp}) instead of one general wip folder

    • combination of different options

      • easybuild/easyconfigs (upstream)
      • vsc (installed in VSC sites, open PR to upstream)
      • site/{brussels, ghent, leuven, antwerp} (free to push, works but maybe site-specific, needs work to push upstream)
      • wip/ (same structure as in vsc-software-stack currently, linked to issues)

    • EB framework needs a small enhancement to be able to use --from-pr for easyconfigs that are located in a 'vsc' subfolder

      • vsc/easybuild/easyconfigs/{a-z}
      • eb --pr-target-repo vscentrum --pr-repo-subdir vsc --from-pr 12345
      • alias ebvsc='eb --pr-target-repo vscentrum --pr-repo-subdir vsc'
      • ebvsc --from-pr 12345

    • may need to use branches so easyconfigs being moved from one subdir to another are shown in "files changed" tab

  • public repository

    • easier to share with other communities and get help

  • allow but not encourage version divergences

    • use same CI as upstream easyconfigs repo
    • version divergences can happen (e.g. new software with missing version reference upstream)
    • once anything is installed in a cluster it has to be recorded in this library
    • easybuild unit test can be mended to take into account those divergences

  • proper fork of easybuild-easyconfigs under github.com/vscentrum

    • vsc branch with extra dirs 'vsc' + 'wip' (see option B)
    • also use this fork to create PRs upstream (eb --github-user=vscentrum --new-pr)
    • to create PR to VSC fork: eb --github-boegel=YOUR_USER --pr-target-account vscentrum --new-pr
    • upload test report: eb --pr-target-account vscentrum --from-pr 12345 --rebuild --robot --upload-test-report

TODO

  • check ways to set protection rules in github repo per folder (or emulate something similar to that end)

    • site and vsc folders are free-push
    • vsc folder need PRs

  • check ways to handle moving easyconfigs from wip/site folders to vsc folder

    • PRs will not show diff, since it's just a move
    • using branches instead of folders does not allow to easily merge single easyconfigs

  • implement support for EasyBuild configuration option --pr-repo-subdir

    • --new-pr/--from-pr does not know about subfolders outside easybuild/easyconfig

eb --new-pr --pr-target-account vscentrum --pr-repo-subdir vsc --pr-target-branch vsc example.eb => add easyconfig in vsc/easybuild/easyconfigs in vsc branch

  • set up test repo with proposed structure, and play with it

  • check ways to transition vsc-software-stack to a fork of easybuild-easyconfigs

    • rename current vscentrum/vsc-software-stack to vscentrum/vsc-software-stack_ORIG
    • create proper fork of easybuilders/easybuild-easyconfigs to vscentrum/vsc-software-stack
    • move issues from vscentrum/vsc-software-stack_ORIG to vscentrum/vsc-software-stack
    • move WIP stuff from vscentrum/vsc-software-stack_ORIG to vscentrum/vsc-software-stack/wip
Clone this wiki locally