From 73d519995044b0e779ff4daaf8fa75eaeff1a974 Mon Sep 17 00:00:00 2001 From: stdweird Date: Fri, 22 Sep 2023 10:31:27 +0200 Subject: [PATCH 1/7] ewald teaching/trainig doc --- mkdocs/docs/HPC/teaching_training.md | 139 +++++++++++++++++++++++++++ 1 file changed, 139 insertions(+) create mode 100644 mkdocs/docs/HPC/teaching_training.md diff --git a/mkdocs/docs/HPC/teaching_training.md b/mkdocs/docs/HPC/teaching_training.md new file mode 100644 index 00000000000..8e285842e55 --- /dev/null +++ b/mkdocs/docs/HPC/teaching_training.md @@ -0,0 +1,139 @@ +# Teaching and training + +The HPC infrastructure can be used for teaching and training purposes, and +HPC-UGent provides support for getting you organized. + +As a reminder, both Bachelor and Master students are allowed to use the HPC, and it is also possible to organize workshops. But in either case we do recommend preparing a fallback plan for should the HPC infrastructure be unavailable, e.g. because of an unexpected power failure. + +In general, we advise the use of the HPC webportal in combination with the [interactive cluster {https://docs.hpc.ugent.be/interactive_debug/} ] for teaching and training, but deviations are possible upon request. + +In order to prepare things for your course, provide hpc@ugent.be with the following information (explained further below): + +- UGent Course Code or workshop name +- Vsc-ids of all course group moderators +- Start and end date (and time) for your course/workshop +- In the case of a workshop: vsc-ids of all participants +- Optional information + - special storage requirements: shared folder and/or shared subgroup folders + - reservation for resource requirements beyond the interactive cluster + - ticket number for specific software needed for your course/workshop + - more details for a custom Interactive Application in the webportal + +In addition, it could be beneficial to set up a short Teams call with HPC-UGent, especially if you are using a complex workflow for your course/workshop. + +Please make these requests well in advance, several weeks before the start of your course/workshop. + + +## UGent Course Code + +Based on the Course Code, we can create vsc accounts for all students that have officially enrolled in your UGent course (if they do not have an account already). Students will then no longer have to take steps themselves to request a vsc account. The students do need to be officially enrolled, so that they are linked to your Course Code. +The created vsc accounts will be accounts without an ssh-key. This allows the students to use e.g. the portal, but if they require ssh access to the infrastructure, they will have to add an ssh-key themselves (via the [VSC accountpage] {https://account.vscentrum.be}). + +A group will be automatically created for your course, with all vsc accounts of registered students as member. +Typical format ?gcourse__?, e.g. ?gcourse_e071400_2023? +Teachers are moderator(s) of this course group (see further), but won?t be able to add unregistered students or moderators. Vsc accounts that are not linked to the Course Code will be automatically removed from the course group. To get a student added to the course group, make sure that the student becomes officially enrolled in your course. + + +## Workshops + +(Currently under construction:) +For workshops or courses that do not have a Course Code, you need to provide us with the course or workshop name and a list of all vsc-ids. A group will be made, based on the name of the workshop, with all vsc-ids as member. Teachers/moderators will be able to add/remove vsc accounts from this course group. But students will have to follow the [procedure to request a vsc account]{ https://docs.hpc.ugent.be/account/} themselves. There will be no automation. + + +## Group moderators + +The course group is used a.o. to manage storage permissions. Moderators are the owners of a group and have raised privileges: e.g. they have read/write access in specific folders, can assign subgroups, etc. +Typically, the teachers map to the role of moderators of a course group, and students are regular members. +Provide us with a list of all the vsc accounts for the teachers of your course. + + +## Start and end date (and time) for your course/workshop + +The end date is used to automatically perform a cleanup when your course/workshop has finished: +- Course group and subgroups will be deactivated +- Reservations will be terminated +- Residual data in the course directories will be deleted +- Custom Interactive Applications will be disabled + +The start date (and time) is used as a target for the HPC-UGent team to set up your course requirements. But note that this target is best-effort, depending on the load of the support team and the complexity of your requirements. +Requests should be made well in advance, at least several weeks before the actual start of your course. The sooner you make the request, the better. + + +## Dedicated course storage + +For every course, a dedicated course directory will be created on the DATA filesystem under: /data/gent/courses//_ +e.g. /data/gent/courses/2023/e071400_2023 + +This directory will be accessible by all members of your course group. (Hence, it is no longer necessary to set up or invite course members to your virtual organization.) +Every course directory will always contain the folders: +- /input + - ideally suited to distribute initial common datasets or input data + - moderators have read/write access + - group members (students) only have read access +- /members + - this directory contains a personal folder for every student in your course: + /members/vsc<4abcd> + - only this specific vsc-id will have read/write access to this folder + - moderators have read access to this folder + +Optionally, we can also create these folders: +- /shared + - this is a folder for sharing files between any and all group members + - all group members and moderators have read/write access + - beware that group members will be able to alter/delete each others files in this folder +- /subgroups/group[abc] + - these folders are suitable if you want to let your students collaborate closely in smaller groups + - a number of ?group[abc]? folders are created under the subgroups folder + - each of these group[abc] folders are owned by a dedicated group + - teachers are automatically made moderators of these dedicated groups + - moderators can populate these groups with vsc-ids of group members in the VSC accountpage + - only these vsc-ids will then be able to access a group[abc] folder, and will have read/write access + +If you need any of these additional folders, do indicate under Optional storage requirements: +/shared YES-NO +/subgroups HOW MANY SUBGROUPS + +The data policy for the dedicated course storage is the following: on the indicated end date of your course, the course directory will be made unaccessible to all. One year after the end date it will be permanently removed. We assume that teachers/moderators always have an own copy of the course data as a starting point for a next course. + + +## Resource requirements beyond the interactive cluster + +We assume that your course requirements are such that the [interactive cluster {https://docs.hpc.ugent.be/interactive_debug/} ] can be used. If these resources are insufficient, you will need to request and motivate a reservation. + +Indicate which cluster you would need and the number of nodes, cores, GPUs. +Also clearly indicate when you would need these resources, i.e. the dates and times of each course session. + +Be aware that students will have no access to the reservation outside the course sessions. +This might be relevant when requesting a custom application. + +Reservations take away precious resources for all HPC users, so only request this when it is really needed for your course. In our experience, the [interactive cluster {https://docs.hpc.ugent.be/interactive_debug/} ] is more than sufficient for the majority of cases. + + +## Specific software + +In case you need software for your course/workshop that is unavailable or that needs to be updated, make a separate [software installation request {https://www.ugent.be/hpc/en/support/software-installation-request } ]. Mail hpc@ugent.be the OTRS ticket number in your request for a course/workshop. + +We will try to make the software available before the start of your course/workshop, but this is best effort, depending on the load of the support team and the complexity of your software request. Typically, software installation requests must be made at least one month before the course/workshop starts. + +Ideally, courses/workshops rely on software that is already in use (and thus also well tested). + + +## Custom Interactive Application in the webportal + +HPC-UGent can create a custom interactive application in the web portal for your course/workshop. Typically, this is a generic interactive application such as cluster desktop, Jupyter notebook, ? in which a number of options are preset or locked down: e.g. the number of cores, software version, cluster selection, autostart code, etc. This could make it easier for teachers and students, since students are less prone to making mistakes and don?t have to spend time copy-pasting specific settings. + +A custom interactive application will only be available to the members of your course group. It will appear in the ?Interactive Apps? menu in the webportal, under the section ?Courses?. After the indicated end date of your course, this application will be removed. + +If you would like this for your course, provide hpc@ugent.be with more details, including: +- what interactive application would you like to get launched (cluster desktop, Jupyter, ?) +- which cluster you use +- how many nodes/cores/GPUs are needed +- which software modules you are loading +- custom code you are launching (e.g. in a terminal) +- required environment variables that you are setting +- standard +- ? + +We will try to make the custom interactive application available before the start of your course/workshop, but this is best effort, depending on the load of the support team and the complexity of your request. + +A caveat for the teacher and students is that students don't learn to work with the generic application, and don?t see the actual commands or customization code. Therefore, per custom interactive application, HPC-UGent will make a dedicated section in the [web portal chapter of the HPC user documentation {https://docs.hpc.ugent.be/web_portal/#custom-apps} ]. This section will briefly explain what happens under the hood of the interactive application. We would recommend that you as a teacher take some time to show and explain this to the students. Note that the custom interactive application will disappear for students after the indicated end of your course, but the section in the web portal will remain there for several years, for reference. From 394b19d72f28ea1131028c29cd32d05586f8638f Mon Sep 17 00:00:00 2001 From: stdweird Date: Fri, 22 Sep 2023 10:32:35 +0200 Subject: [PATCH 2/7] add teaching page to index --- config/templates/hpc.template | 1 + 1 file changed, 1 insertion(+) diff --git a/config/templates/hpc.template b/config/templates/hpc.template index 75923f4b516..5e3a1807199 100644 --- a/config/templates/hpc.template +++ b/config/templates/hpc.template @@ -34,6 +34,7 @@ nav: {%- if site == 'Gent' %} - GPU clusters: gpu.md - Cron scripts: crontab.md + - Teaching and training: teaching_training.md - Linux tutorial: - Introduction: linux-tutorial/index.md - Getting Started: linux-tutorial/getting_started.md From 548e91140013d64445f81814fdaec0043620d06a Mon Sep 17 00:00:00 2001 From: stdweird Date: Fri, 29 Sep 2023 14:46:36 +0200 Subject: [PATCH 3/7] first review --- mkdocs/docs/HPC/teaching_training.md | 192 +++++++++++++++++---------- 1 file changed, 125 insertions(+), 67 deletions(-) diff --git a/mkdocs/docs/HPC/teaching_training.md b/mkdocs/docs/HPC/teaching_training.md index 8e285842e55..78115a96005 100644 --- a/mkdocs/docs/HPC/teaching_training.md +++ b/mkdocs/docs/HPC/teaching_training.md @@ -1,53 +1,53 @@ # Teaching and training The HPC infrastructure can be used for teaching and training purposes, and -HPC-UGent provides support for getting you organized. +HPC-UGent provides support for getting you organized. -As a reminder, both Bachelor and Master students are allowed to use the HPC, and it is also possible to organize workshops. But in either case we do recommend preparing a fallback plan for should the HPC infrastructure be unavailable, e.g. because of an unexpected power failure. +As a reminder, both Bachelor and Master students are allowed to use the HPC infrastructure, +and it is also possible to organize trainings (or workshops). +But in either case we do recommend preparing a fallback plan in case the HPC infrastructure becomes unavailable, +e.g. because of an unexpected power failure. -In general, we advise the use of the HPC webportal in combination with the [interactive cluster {https://docs.hpc.ugent.be/interactive_debug/} ] for teaching and training, but deviations are possible upon request. +In general, we advise the use of the HPC webportal in combination with the +[interactive cluster](interactive_debug.md) +for teaching and training, but deviations are possible upon request. -In order to prepare things for your course, provide hpc@ugent.be with the following information (explained further below): +In order to prepare things, make a _teaching request_ by [contacting the {{ hpcteam }}]({{ hpc_support_url }}) +with the following information (explained further below): -- UGent Course Code or workshop name -- Vsc-ids of all course group moderators -- Start and end date (and time) for your course/workshop -- In the case of a workshop: vsc-ids of all participants +- Title and nickname +- VSC-ids of all course teachers/trainers +- Start and end date for your course or training +- UGent Course Code and/or VSC-ids to create list of participants - Optional information - - special storage requirements: shared folder and/or shared subgroup folders + - special storage requirements + - shared folder + - quota requirements + - subgroups for team collaboration - reservation for resource requirements beyond the interactive cluster - - ticket number for specific software needed for your course/workshop + - ticket number for specific software needed for your course/training - more details for a custom Interactive Application in the webportal -In addition, it could be beneficial to set up a short Teams call with HPC-UGent, especially if you are using a complex workflow for your course/workshop. +In addition, it could be beneficial to set up a short Teams call with HPC-UGent team members, +especially if you are using a complex workflow for your course/workshop. Please make these requests well in advance, several weeks before the start of your course/workshop. -## UGent Course Code +## Title and nickname -Based on the Course Code, we can create vsc accounts for all students that have officially enrolled in your UGent course (if they do not have an account already). Students will then no longer have to take steps themselves to request a vsc account. The students do need to be officially enrolled, so that they are linked to your Course Code. -The created vsc accounts will be accounts without an ssh-key. This allows the students to use e.g. the portal, but if they require ssh access to the infrastructure, they will have to add an ssh-key themselves (via the [VSC accountpage] {https://account.vscentrum.be}). -A group will be automatically created for your course, with all vsc accounts of registered students as member. -Typical format ?gcourse__?, e.g. ?gcourse_e071400_2023? -Teachers are moderator(s) of this course group (see further), but won?t be able to add unregistered students or moderators. Vsc accounts that are not linked to the Course Code will be automatically removed from the course group. To get a student added to the course group, make sure that the student becomes officially enrolled in your course. +## Teachers and/or trainers - -## Workshops - -(Currently under construction:) -For workshops or courses that do not have a Course Code, you need to provide us with the course or workshop name and a list of all vsc-ids. A group will be made, based on the name of the workshop, with all vsc-ids as member. Teachers/moderators will be able to add/remove vsc accounts from this course group. But students will have to follow the [procedure to request a vsc account]{ https://docs.hpc.ugent.be/account/} themselves. There will be no automation. - - -## Group moderators - -The course group is used a.o. to manage storage permissions. Moderators are the owners of a group and have raised privileges: e.g. they have read/write access in specific folders, can assign subgroups, etc. +Teacher are Group moderators +The course group is used a.o. to manage storage permissions. +Moderators are the owners of a group and have raised privileges: +e.g. they have read/write access in specific folders, can assign subgroups, etc. Typically, the teachers map to the role of moderators of a course group, and students are regular members. -Provide us with a list of all the vsc accounts for the teachers of your course. +Provide us with a list of all the VSC-ids for the teachers of your course. -## Start and end date (and time) for your course/workshop +## Start and end date for your course/training The end date is used to automatically perform a cleanup when your course/workshop has finished: - Course group and subgroups will be deactivated @@ -55,85 +55,143 @@ The end date is used to automatically perform a cleanup when your course/worksho - Residual data in the course directories will be deleted - Custom Interactive Applications will be disabled -The start date (and time) is used as a target for the HPC-UGent team to set up your course requirements. But note that this target is best-effort, depending on the load of the support team and the complexity of your requirements. +The start date (and time) is used as a target for the HPC-UGent team to set up your course requirements. But note that this target is best-effort, depending on the load of the support team and the complexity of your requirements. Requests should be made well in advance, at least several weeks before the actual start of your course. The sooner you make the request, the better. +## Participants + +### UGent Cuurses + +Based on the Course Code, we can create VSC accounts for all UGent students that have officially enrolled +in your UGent course (if they do not have an account already). +Students will then no longer have to take steps themselves to request a VSC account. +The students do need to be officially enrolled, so that they are linked to your UGent Course Code. +The created VSC accounts will be accounts without an ssh-key. +This allows the students to use e.g. the portal, but if they require ssh access to the infrastructure, +they will have to [add an SSH key](account.md#adding-multiple-ssh-public-keys-optional) themselves. + +A group will be automatically created for your course, with all VSC accounts of registered students as member. +Typical format `gcourse__`, e.g. `gcourse_e071400_2023`. +Teachers are moderator(s) of this course group (see further), but will not be able to add unregistered students or moderators. +VSC accounts that are not linked to the Course Code will be automatically removed from the course group. +To get a student added to the course group, make sure that the student becomes officially enrolled in your course. + + +### Trainings and workshops + +(Currently under construction:) +For trainings, workshops or courses that do not have a Course Code, you need to provide us with the list of all VSC-ids. +A group will be made, based on the name of the workshop, with all VSC-ids as member. +Teachers/trainers will be able to add/remove VSC accounts from this course group. +But students will have to follow the [procedure to request a VSC account](account.md) themselves. +There will be no automation. + ## Dedicated course storage -For every course, a dedicated course directory will be created on the DATA filesystem under: /data/gent/courses//_ -e.g. /data/gent/courses/2023/e071400_2023 +For every course, a dedicated course directory will be created on the _DATA_ filesystem +under `/data/gent/courses//_` (e.g. `/data/gent/courses/2023/cae_e071400`). + +This directory will be accessible by all members of your course group. +(Hence, it is no longer necessary to set up dangerous workarounds e.g. invite course members to your virtual organization.) -This directory will be accessible by all members of your course group. (Hence, it is no longer necessary to set up or invite course members to your virtual organization.) Every course directory will always contain the folders: -- /input - - ideally suited to distribute initial common datasets or input data + +- `input` + - ideally suited to distribute input data such as common datasets - moderators have read/write access - - group members (students) only have read access -- /members - - this directory contains a personal folder for every student in your course: - /members/vsc<4abcd> - - only this specific vsc-id will have read/write access to this folder + - group members (students) only have read access +- `members` + - this directory contains a personal folder for every student in your course `members/vsc<01234>` + - only this specific VSC-id will have read/write access to this folder - moderators have read access to this folder Optionally, we can also create these folders: -- /shared + +- `shared` - this is a folder for sharing files between any and all group members - all group members and moderators have read/write access - - beware that group members will be able to alter/delete each others files in this folder -- /subgroups/group[abc] + - beware that group members will be able to alter/delete each others files in this folder if they set + permissions in specific/non-default ways +- `subgroups/group<01>` - these folders are suitable if you want to let your students collaborate closely in smaller groups - - a number of ?group[abc]? folders are created under the subgroups folder - - each of these group[abc] folders are owned by a dedicated group + - a number of `group<01>` folders are created under the `subgroups` folder + - each of these `group<01>` folders are owned by a dedicated group - teachers are automatically made moderators of these dedicated groups - - moderators can populate these groups with vsc-ids of group members in the VSC accountpage - - only these vsc-ids will then be able to access a group[abc] folder, and will have read/write access + - moderators can populate these groups with VSC-ids of group members in the VSC accountpage + or ask the students to invite themself (moderators still need to approve these invites). + - only these VSC-ids will then be able to access a `group<01>` folder, and will have read/write access. If you need any of these additional folders, do indicate under Optional storage requirements: -/shared YES-NO -/subgroups HOW MANY SUBGROUPS +`shared` `yes` +`subgroups` `number of subgroups` -The data policy for the dedicated course storage is the following: on the indicated end date of your course, the course directory will be made unaccessible to all. One year after the end date it will be permanently removed. We assume that teachers/moderators always have an own copy of the course data as a starting point for a next course. +The data policy for the dedicated course storage is the following: +on the indicated end date of your course, the course directory will be made read-only to the moderators +(possibly on the form of an archive zipfile). +One year after the end date it will be permanently removed. +We assume that teachers/trainers always have an own copy of the course data as a starting point for a next course. ## Resource requirements beyond the interactive cluster -We assume that your course requirements are such that the [interactive cluster {https://docs.hpc.ugent.be/interactive_debug/} ] can be used. If these resources are insufficient, you will need to request and motivate a reservation. +We assume that your course requirements are such that the [interactive cluster](interactive_debug.md) can be used. +If these resources are insufficient, you will need to request and motivate a reservation. -Indicate which cluster you would need and the number of nodes, cores, GPUs. +Indicate which cluster you would need and the number of nodes, cores and/or GPUs. Also clearly indicate when you would need these resources, i.e. the dates and times of each course session. Be aware that students will have no access to the reservation outside the course sessions. This might be relevant when requesting a custom application. -Reservations take away precious resources for all HPC users, so only request this when it is really needed for your course. In our experience, the [interactive cluster {https://docs.hpc.ugent.be/interactive_debug/} ] is more than sufficient for the majority of cases. +Reservations take away precious resources for all HPC users, so only request this when it is really needed for your course. +In our experience, the [interactive cluster](interactive_debug.md) is more than sufficient for the majority of cases. ## Specific software -In case you need software for your course/workshop that is unavailable or that needs to be updated, make a separate [software installation request {https://www.ugent.be/hpc/en/support/software-installation-request } ]. Mail hpc@ugent.be the OTRS ticket number in your request for a course/workshop. +In case you need software for your course/workshop that is unavailable or that needs to be updated, +make a separate [software installation request](https://www.ugent.be/hpc/en/support/software-installation-request). +Add the OTRS ticket number in your _teaching request_. -We will try to make the software available before the start of your course/workshop, but this is best effort, depending on the load of the support team and the complexity of your software request. Typically, software installation requests must be made at least one month before the course/workshop starts. +We will try to make the software available before the start of your course/workshop. +But this is always best effort, depending on the load of the support team and the complexity of your software request. +Typically, software installation requests must be made at least one month before the course/workshop starts. Ideally, courses/workshops rely on software that is already in use (and thus also well tested). ## Custom Interactive Application in the webportal -HPC-UGent can create a custom interactive application in the web portal for your course/workshop. Typically, this is a generic interactive application such as cluster desktop, Jupyter notebook, ? in which a number of options are preset or locked down: e.g. the number of cores, software version, cluster selection, autostart code, etc. This could make it easier for teachers and students, since students are less prone to making mistakes and don?t have to spend time copy-pasting specific settings. -A custom interactive application will only be available to the members of your course group. It will appear in the ?Interactive Apps? menu in the webportal, under the section ?Courses?. After the indicated end date of your course, this application will be removed. +HPC-UGent can create a custom interactive application in the web portal for your course/workshop. +Typically, this is a generic interactive application such as cluster desktop, Jupyter notebook, ... +in which a number of options are preset or locked down: +e.g. the number of cores, software version, cluster selection, autostart code, etc. +This could make it easier for teachers and students, since students are less prone to making mistakes +and do not have to spend time copy-pasting specific settings. + +A custom interactive application will only be available to the members of your course group. +It will appear in the `Interactive Apps` menu in the webportal, under the section `Courses`. +After the indicated end date of your course, this application will be removed. -If you would like this for your course, provide hpc@ugent.be with more details, including: -- what interactive application would you like to get launched (cluster desktop, Jupyter, ?) -- which cluster you use +If you would like this for your course, provide more details in your _teaching request_, including: +- what interactive application would you like to get launched (cluster desktop, Jupyter, ...) +- which cluster you want to use - how many nodes/cores/GPUs are needed - which software modules you are loading -- custom code you are launching (e.g. in a terminal) +- custom code you are launching (e.g. autostart a GUI) - required environment variables that you are setting -- standard -- ? - -We will try to make the custom interactive application available before the start of your course/workshop, but this is best effort, depending on the load of the support team and the complexity of your request. - -A caveat for the teacher and students is that students don't learn to work with the generic application, and don?t see the actual commands or customization code. Therefore, per custom interactive application, HPC-UGent will make a dedicated section in the [web portal chapter of the HPC user documentation {https://docs.hpc.ugent.be/web_portal/#custom-apps} ]. This section will briefly explain what happens under the hood of the interactive application. We would recommend that you as a teacher take some time to show and explain this to the students. Note that the custom interactive application will disappear for students after the indicated end of your course, but the section in the web portal will remain there for several years, for reference. +- ... + +We will try to make the custom interactive application available before the start of your course/workshop, +but this is always best effort, depending on the load of the support team and the complexity of your request. + +A caveat for the teacher and students is that students do not learn to work with the generic application, +and do not see the actual commands or customization code. +Therefore, per custom interactive application, HPC-UGent will make a dedicated section in the +[web portal chapter of the HPC user documentation](webportal.md#custom-apps). +This section will briefly explain what happens under the hood of the interactive application. +We would recommend that you as a teacher take some time to show and explain this to the students. +Note that the custom interactive application will disappear for students after the indicated end of your course, +but the section in the web portal will remain there for several years, for reference. From 3dff4087006a7728e51db2a8eeab22818f42e0b5 Mon Sep 17 00:00:00 2001 From: stdweird Date: Fri, 29 Sep 2023 14:46:50 +0200 Subject: [PATCH 4/7] fixes to build/install process --- build.sh | 17 +++++++++-------- requirements.txt | 1 + 2 files changed, 10 insertions(+), 8 deletions(-) diff --git a/build.sh b/build.sh index db3a37f837f..1f09ea17683 100755 --- a/build.sh +++ b/build.sh @@ -5,18 +5,19 @@ if [ "${DEBUG:-0}" -gt 0 ]; then fi DEPSDIR="$PWD/pypkgs" -if [ "${INSTALLDEPS:-0}" -gt 0 ]; then - python3 -m pip install --ignore-installed --prefix "$DEPSDIR" -r requirements.txt - python3 -m pip install --prefix "$DEPSDIR" "$PWD"/custom_plugin - python3 -m pip install --prefix "$DEPSDIR" "$PWD"/computational_macros - # not needed for actual page - python3 -m pip install --ignore-installed --prefix "$DEPSDIR" linkchecker -fi - pyver=python$(python3 -c "import sys; print(\"%s.%s\" % sys.version_info[:2])") export PYTHONPATH=$DEPSDIR/lib/$pyver/site-packages:$DEPSDIR/lib64/$pyver/site-packages:$PWD/custom_plugin:$PWD/computational_macros:$PYTHONPATH export PATH=$DEPSDIR/bin:$PATH + +if [ "${INSTALLDEPS:-0}" -gt 0 ]; then + python3 -m pip install --use-pep517 --ignore-installed --prefix "$DEPSDIR" -r requirements.txt + python3 -m pip install --use-pep517 --prefix "$DEPSDIR" "$PWD"/custom_plugin + python3 -m pip install --use-pep517 --prefix "$DEPSDIR" "$PWD"/computational_macros + # not needed for actual page + python3 -m pip install --use-pep517 --ignore-installed --prefix "$DEPSDIR" linkchecker +fi + python3 build.py "$@" if [ "${WEBSERVER:-0}" -gt 0 ]; then diff --git a/requirements.txt b/requirements.txt index cdb52ef2f09..94acb4fc2a8 100644 --- a/requirements.txt +++ b/requirements.txt @@ -30,3 +30,4 @@ symspellpy==6.7.7 termcolor==2.2.0 watchdog==3.0.0 zipp==3.15.0 +setuptools==68.2.2 From 88b68e14038ae11989eed3b466c299becbddc3e8 Mon Sep 17 00:00:00 2001 From: stdweird Date: Fri, 29 Sep 2023 14:56:44 +0200 Subject: [PATCH 5/7] fix broken link and long lines --- mkdocs/docs/HPC/teaching_training.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/mkdocs/docs/HPC/teaching_training.md b/mkdocs/docs/HPC/teaching_training.md index 78115a96005..a7a281b8ca3 100644 --- a/mkdocs/docs/HPC/teaching_training.md +++ b/mkdocs/docs/HPC/teaching_training.md @@ -55,8 +55,10 @@ The end date is used to automatically perform a cleanup when your course/worksho - Residual data in the course directories will be deleted - Custom Interactive Applications will be disabled -The start date (and time) is used as a target for the HPC-UGent team to set up your course requirements. But note that this target is best-effort, depending on the load of the support team and the complexity of your requirements. -Requests should be made well in advance, at least several weeks before the actual start of your course. The sooner you make the request, the better. +The start date (and time) is used as a target for the HPC-UGent team to set up your course requirements. +But note that this target is best-effort, depending on the load of the support team and the complexity of your requirements. +Requests should be made well in advance, at least several weeks before the actual start of your course. +The sooner you make the request, the better. ## Participants @@ -190,7 +192,7 @@ but this is always best effort, depending on the load of the support team and th A caveat for the teacher and students is that students do not learn to work with the generic application, and do not see the actual commands or customization code. Therefore, per custom interactive application, HPC-UGent will make a dedicated section in the -[web portal chapter of the HPC user documentation](webportal.md#custom-apps). +[web portal chapter of the HPC user documentation](web_portal.md#custom-apps). This section will briefly explain what happens under the hood of the interactive application. We would recommend that you as a teacher take some time to show and explain this to the students. Note that the custom interactive application will disappear for students after the indicated end of your course, From 5860e8d99148cd24cbaf0b1323b39f9bcd629748 Mon Sep 17 00:00:00 2001 From: stdweird Date: Fri, 29 Sep 2023 15:06:02 +0200 Subject: [PATCH 6/7] nice gent typo --- mkdocs/docs/HPC/teaching_training.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mkdocs/docs/HPC/teaching_training.md b/mkdocs/docs/HPC/teaching_training.md index a7a281b8ca3..8110659d569 100644 --- a/mkdocs/docs/HPC/teaching_training.md +++ b/mkdocs/docs/HPC/teaching_training.md @@ -62,7 +62,7 @@ The sooner you make the request, the better. ## Participants -### UGent Cuurses +### UGent Courses Based on the Course Code, we can create VSC accounts for all UGent students that have officially enrolled in your UGent course (if they do not have an account already). From 19774f0c4415a18d0fc99c67a12fe8fdd8b57fd6 Mon Sep 17 00:00:00 2001 From: stdweird Date: Fri, 29 Sep 2023 17:57:17 +0200 Subject: [PATCH 7/7] more rewording/cleanup --- mkdocs/docs/HPC/teaching_training.md | 137 +++++++++++++++++---------- 1 file changed, 86 insertions(+), 51 deletions(-) diff --git a/mkdocs/docs/HPC/teaching_training.md b/mkdocs/docs/HPC/teaching_training.md index 8110659d569..492198ddd37 100644 --- a/mkdocs/docs/HPC/teaching_training.md +++ b/mkdocs/docs/HPC/teaching_training.md @@ -16,17 +16,17 @@ In order to prepare things, make a _teaching request_ by [contacting the {{ hpct with the following information (explained further below): - Title and nickname -- VSC-ids of all course teachers/trainers - Start and end date for your course or training -- UGent Course Code and/or VSC-ids to create list of participants +- VSC-ids of all teachers/trainers +- Participants based on UGent Course Code and/or list of VSC-ids - Optional information - - special storage requirements - - shared folder - - quota requirements - - subgroups for team collaboration - - reservation for resource requirements beyond the interactive cluster - - ticket number for specific software needed for your course/training - - more details for a custom Interactive Application in the webportal + - Additional storage requirements + - Shared folder + - Groups folder for collaboration + - Quota + - Reservation for resource requirements beyond the interactive cluster + - Ticket number for specific software needed for your course/training + - Details for a custom Interactive Application in the webportal In addition, it could be beneficial to set up a short Teams call with HPC-UGent team members, especially if you are using a complex workflow for your course/workshop. @@ -36,49 +36,65 @@ Please make these requests well in advance, several weeks before the start of yo ## Title and nickname +The title of the course or training can be used in e.g. reporting. -## Teachers and/or trainers +The nickname is a single (short) word or acronym that the students or participants can easily +recognise, e.g. in the directory structure. In case of UGent courses, this is used next to the +course code to help identify [the course directory](./#dedicated-storage) in the list of all courses one might follow. -Teacher are Group moderators -The course group is used a.o. to manage storage permissions. -Moderators are the owners of a group and have raised privileges: -e.g. they have read/write access in specific folders, can assign subgroups, etc. -Typically, the teachers map to the role of moderators of a course group, and students are regular members. -Provide us with a list of all the VSC-ids for the teachers of your course. +When choosing the nickname, try to make it unique, but this is not enforced nor checked. - -## Start and end date for your course/training - -The end date is used to automatically perform a cleanup when your course/workshop has finished: -- Course group and subgroups will be deactivated -- Reservations will be terminated -- Residual data in the course directories will be deleted -- Custom Interactive Applications will be disabled +## Start and end date The start date (and time) is used as a target for the HPC-UGent team to set up your course requirements. But note that this target is best-effort, depending on the load of the support team and the complexity of your requirements. Requests should be made well in advance, at least several weeks before the actual start of your course. The sooner you make the request, the better. +The end date is used to automatically perform a cleanup when your course/workshop has finished, +as described in the [course data policy](./#course-data-policy): + +- Course group and subgroups will be deactivated +- Residual data in the course directories will be archived or deleted +- Custom Interactive Applications will be disabled + + +## Teachers and trainers + +A _course group_ is created with all students or participants, and the teachers or trainers are +the group moderators (and also member of this group). + +This course group and the moderators group are used to manage the different priviliges: +moderators have additional privileges over non-moderator members +e.g. they have read/write access in specific folders, can manage subgroups, .... + +Provide us with a list of all the VSC-ids for the teachers or trainers to indentify the moderators. + + ## Participants +The management of the list of students or participants depends if this is a UGent course or a training/workshop. + ### UGent Courses Based on the Course Code, we can create VSC accounts for all UGent students that have officially enrolled in your UGent course (if they do not have an account already). Students will then no longer have to take steps themselves to request a VSC account. The students do need to be officially enrolled, so that they are linked to your UGent Course Code. + The created VSC accounts will be accounts without an ssh-key. This allows the students to use e.g. the portal, but if they require ssh access to the infrastructure, they will have to [add an SSH key](account.md#adding-multiple-ssh-public-keys-optional) themselves. -A group will be automatically created for your course, with all VSC accounts of registered students as member. +Additionally, for external, non-UGent students the _teaching request_ must contain the list +of their VSC-ids, so they can be added to the course group. + +A course group will be automatically created for your course, with all VSC accounts of registered students as member. Typical format `gcourse__`, e.g. `gcourse_e071400_2023`. -Teachers are moderator(s) of this course group (see further), but will not be able to add unregistered students or moderators. +Teachers are [moderator of this course group](./#teachers-and-trainers), but will not be able to add unregistered students or moderators. VSC accounts that are not linked to the Course Code will be automatically removed from the course group. To get a student added to the course group, make sure that the student becomes officially enrolled in your course. - ### Trainings and workshops (Currently under construction:) @@ -89,7 +105,7 @@ But students will have to follow the [procedure to request a VSC account](accoun There will be no automation. -## Dedicated course storage +## Dedicated storage For every course, a dedicated course directory will be created on the _DATA_ filesystem under `/data/gent/courses//_` (e.g. `/data/gent/courses/2023/cae_e071400`). @@ -100,33 +116,52 @@ This directory will be accessible by all members of your course group. Every course directory will always contain the folders: - `input` - - ideally suited to distribute input data such as common datasets - - moderators have read/write access - - group members (students) only have read access + - ideally suited to distribute input data such as common datasets + - moderators have read/write access + - group members (students) only have read access - `members` - - this directory contains a personal folder for every student in your course `members/vsc<01234>` - - only this specific VSC-id will have read/write access to this folder - - moderators have read access to this folder + - this directory contains a personal folder for every student in your course `members/vsc<01234>` + - only this specific VSC-id will have read/write access to this folder + - moderators have read access to this folder + + +### Shared and groups Optionally, we can also create these folders: - `shared` - - this is a folder for sharing files between any and all group members - - all group members and moderators have read/write access - - beware that group members will be able to alter/delete each others files in this folder if they set - permissions in specific/non-default ways -- `subgroups/group<01>` - - these folders are suitable if you want to let your students collaborate closely in smaller groups - - a number of `group<01>` folders are created under the `subgroups` folder - - each of these `group<01>` folders are owned by a dedicated group - - teachers are automatically made moderators of these dedicated groups - - moderators can populate these groups with VSC-ids of group members in the VSC accountpage - or ask the students to invite themself (moderators still need to approve these invites). - - only these VSC-ids will then be able to access a `group<01>` folder, and will have read/write access. - -If you need any of these additional folders, do indicate under Optional storage requirements: -`shared` `yes` -`subgroups` `number of subgroups` + - this is a folder for sharing files between any and all group members + - all group members and moderators have read/write access + - beware that group members will be able to alter/delete each others files in this folder if they set + permissions in specific/non-default ways +- `groups` + - a number of `groups/group_<01>` folders are created under the `groups` folder + - these folders are suitable if you want to let your students collaborate closely in smaller groups + - each of these `group_<01>` folders are owned by a dedicated group + - teachers are automatically made moderators of these dedicated groups + - moderators can populate these groups with VSC-ids of group members in the VSC accountpage + or ask the students to invite themself via [group edit](https://account.vscentrum.be/django/group/edit). + When students invite them self, moderators still need to [approve the group invites](https://account.vscentrum.be/django/group/approve). + - only these VSC-ids will then be able to access a `group_<01>` folder, and will have read/write access. + + +If you need any of these additional folders, do indicate under _Optional storage requirements_ +of your _teaching request_: + +- `shared`: `yes` +- `subgroups`: `` + +### Course Quota + +There are 4 quota settings that you can choose in your _teaching request_ in the case the defaults are not sufficient: + +- overall quota (defaults *10 GB volume* and *20k files*) are for the moderators and can be used for e.g. the `input` folder. +- member quota (defaults *5 GB volume* and *10k files*) are per student/participant + +The course data usage is not accounted for the any other quota (like VO quota). It is solely dependant on these settings. + + +### Course data policy The data policy for the dedicated course storage is the following: on the indicated end date of your course, the course directory will be made read-only to the moderators @@ -178,7 +213,7 @@ It will appear in the `Interactive Apps` menu in the webportal, under the sectio After the indicated end date of your course, this application will be removed. If you would like this for your course, provide more details in your _teaching request_, including: -- what interactive application would you like to get launched (cluster desktop, Jupyter, ...) +- what interactive application would you like to get launched (cluster desktop, Jupyter Notebook, ...) - which cluster you want to use - how many nodes/cores/GPUs are needed - which software modules you are loading