Merge pull request #680 from lbarraga/module-auto-pr

Automatic pull request script: available modules

scripts/modules_auto_pr/README.md

@@ -0,0 +1,61 @@
+# Auto Update Modules Script
+
+This script automates the process of updating markdown files in the `vsc_user_docs` repository by 
+
+- cloning the `hpcugent/vsc_user_docs` repository
+- creating a new branch
+- running the `available_software.py` script
+- committing the changes
+- pushing the changes to a forked repository
+- creating a pull request to the `hpcugent/vsc_user_docs` repository from the forked repository using a GitHub Personal Access Token (PAT)
+
+## Assumptions of the script
+
+The script assumes the following:
+
+- The user who owns the fork has a GitHub personal access token (PAT) with the `repo` and `read:org` scopes. This is used to create the pull request.
+- SSH keys need to be present for the GitHub account that owns the fork. This will be used to push the changed to the fork. This is different from the PAT, which is only used to create the pull request.
+- Lmod is installed. This is used by the `available_software.py` script.
+- The GitHub CLI (`gh`) is installed to create the PR. See [the GitHub CLI installation instructions](https://github.com/cli/cli#installation) for more information.
+- the fork is named `vsc_user_docs`
+
+## Usage
+
+The script should be called with a file containing a
+[GitHub classic personal access token (PAT)](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#personal-access-tokens-classic) 
+and the GitHub name of the user which owns the fork as arguments:
+
+```shell
+$ ./modules_auto_pr.sh path/to/github/PAT <github_username>
+```
+
+## Install the github CLI
+
+On the HPC systems, it is best to download a precompiled binary from the 
+[GitHub CLI releases page](https://github.com/cli/cli/releases).
+Look for the download marked amd64, download, unpack and update `$PATH`.
+
+```shell
+$ gh_version="2.55.0"
+$ curl -OL https://github.com/cli/cli/releases/download/v${gh_version}/gh_${gh_version}_linux_amd64.tar.gz
+$ tar xfvz gh_*_linux_amd64.tar.gz
+$ export PATH=$PATH:$(ls -d $PWD/gh_*_linux_amd64/bin)
+```
+
+## Make a classic Personal Access Token
+
+To make a classic PAT, navigate to the GitHub account setting page (not the repository settings) and go to:
+
+https://github.com/settings/tokens
+
+click on `Generate new token` and fill in the note field and expiration date. 
+
+Under `select scopes`, click on `repo` (which will automatically select all the sub-scopes) and `read:org` (which is a sub-scope of `admin:org`).
+
+Click generate token and copy-paste the token into the file that is passed to the `modules_auto_pr.sh` script (see above)
+
+The reason fine-grained tokens are not used is that they do not yet support writing to a public repository 
+that is not owned by you or an organization that you are not a member of.
+
+
+

scripts/modules_auto_pr/modules_auto_pr.sh

@@ -0,0 +1,98 @@
+#!/bin/bash
+
+set -e  # Exit immediately if a command exits with a non-zero status
+set -u  # Treat unset variables as an error when substituting
+
+####################################################################################################
+#                                            VARIABLES                                             #
+####################################################################################################
+
+TIMESTAMP=$(date +"%Y-%m-%d_%H-%M")
+HUMAN_TIMESTAMP=$(date "+%-d %B %Y, %H:%M")
+REPO_URL="[email protected]:hpcugent/vsc_user_docs" # The repository for which the script will generate the PR
+BASE_BRANCH="main"
+BRANCH_NAME="auto_update_modules_$TIMESTAMP"
+REPO_NAME="vsc_user_docs" # name of the cloned repo. script available_modules.py requires this to be the name. Do not change.
+REPO_PATH="/tmp/$USER/modules_auto_pr_script_$TIMESTAMP/$REPO_NAME"
+COMMIT_MESSAGE="Update Modules [$HUMAN_TIMESTAMP]"
+PR_TITLE="Auto Update Modules [$HUMAN_TIMESTAMP]"
+
+make_pr_body() {
+  local n_added_modules="$1"
+  local n_removed_modules="$2"
+
+  echo "This is an automated pull request updating the markdown files of all available modules."
+  echo ""
+  echo "Changes:"
+  echo "- Updated the markdown files of all previously available modules"
+  echo "- Added $n_added_modules new modules"
+  echo "- Removed $n_removed_modules modules"
+}
+
+####################################################################################################
+#                                             SCRIPT                                               #
+####################################################################################################
+
+echo_info()  { echo -e "\e[32m$0: [INFO] $1\e[0m"; }
+echo_error() { echo -e "\e[31m$0: [ERROR] $1\e[0m"; }
+
+main() {
+  local github_pat_file="$1"
+  local fork_user="$2" # The user that owns the fork from which the script will create the PR. e.g. lbarraga
+  local fork_url="[email protected]:$fork_user/vsc_user_docs" # The fork from which the script will create the PR
+
+  # Check if the `gh` command is available
+  if ! command -v gh &> /dev/null; then
+      echo_error "The GitHub CLI (gh) is not installed."
+      echo_error "Please install it by following the instructions at https://cli.github.com."
+      exit 1
+  fi
+
+  # Check if PAT file is present and fork user is provided in one if statement
+  # -s checks whether file exists and has non-zero size (is not empty)
+  if [ ! -s "$github_pat_file" ] || [ -z "$fork_user" ]; then
+      echo_error "Please provide a valid GitHub Personal Access Token (PAT) file and fork user."
+      echo_error "Usage: $0 <github_pat_file> <fork_user>"
+      exit 1
+  fi
+
+  gh auth login --with-token < "$github_pat_file"
+
+  # Clone the repository and create a new branch
+  git clone $REPO_URL "$REPO_PATH"
+  cd "$REPO_PATH"
+  git checkout -b "$BRANCH_NAME"
+
+  # run available_software.py
+  python -m venv venv
+  source venv/bin/activate
+  pip install -r scripts/available_software/requirements.txt
+  python scripts/available_software/available_software.py
+
+  # Add and commit the generated files
+  git add .
+  git commit -m "$COMMIT_MESSAGE"
+
+  # Calculate the number of added and removed modules
+  N_ADDED_MODULES=$(git show --name-status HEAD | grep -e "^A.*\.md$" | wc -l)
+  N_REMOVED_MODULES=$(git show --name-status HEAD | grep -e "^D.*\.md$" | wc -l)
+
+  # Push the new branch to GitHub
+  git remote add fork "$fork_url"
+  git push fork "$BRANCH_NAME"
+
+  # Set the UGent repo as the default remote
+  gh repo set-default $REPO_URL
+
+  # Create a pull request using GitHub CLI. Pull request is automatically created into the default repository.
+  gh pr create \
+    --title "$PR_TITLE" \
+    --body  "$(make_pr_body "$N_ADDED_MODULES" "$N_REMOVED_MODULES")" \
+    --base  "$BASE_BRANCH" \
+    --head  "$fork_user:$BRANCH_NAME" \
+
+  # Clean up
+  rm -rf "$REPO_PATH"
+}
+
+main "${1:-}" "${2:-}"