From 5c8a12b6640cbdafc15bb83c34ec41247d691759 Mon Sep 17 00:00:00 2001 From: Heather Savoy <47045484+HeatherSavoy-USDA@users.noreply.github.com> Date: Sat, 9 Nov 2024 09:29:48 -0700 Subject: [PATCH 1/5] drafted cv setup instructions --- .../21-computer-vision-2.md | 44 +++++++++++++++++++ .../2024-ai-user-forum/21-computer-vision.md | 42 ++++++++++++++++++ 2 files changed, 86 insertions(+) diff --git a/sn_collections/_workshops/2024-ai-user-forum/21-computer-vision-2.md b/sn_collections/_workshops/2024-ai-user-forum/21-computer-vision-2.md index f11146f4a..6ad1f0da0 100644 --- a/sn_collections/_workshops/2024-ai-user-forum/21-computer-vision-2.md +++ b/sn_collections/_workshops/2024-ai-user-forum/21-computer-vision-2.md @@ -16,3 +16,47 @@ prerequisites: In this workshop, participants will learn the key concepts and techniques needed to use modern, deep learning-based computer vision methods for object detection and semantic segmentation. Learners will practice training and evaluating state-of-the-art computer vision models on custom image datasets. This workshop is intended as a continuation of “Computer vision I: introduction and image classification”, but participants do not need to take the earlier workshop if they already have a basic knowledge of machine learning and computer vision concepts. + + +## Tutorial setup instruction + +Steps to prepare for the tutorial: + +1. **Login to Atlas Open OnDemand** at [https://atlas-ood.hpc.msstate.edu/](https://atlas-ood.hpc.msstate.edu/). For more information on login procedures for web-based SCINet access, see the [SCINet access user guide]({{site.baseurl}}/guides/access/web-based-login). + +1. **Open a command-line session** by clicking on “Clusters” -> “Atlas Shell Access” on the top menu. This will open a new tab with a command-line session on Atlas's login node. + +1. **Create and/or update your workshop working directory** and copy the tutorial materials into it by running the following commands. Note: you do not have to edit the commands with your username as it will be determined by the `$USER` variable. + + {:.copy-code} + ```bash + mkdir -p /90daydata/shared/$USER/computer_vision2 + cd /90daydata/shared/$USER/computer_vision2 + cp -r /project/ai_forum/computer_vision2/computer_vision_2.ipynb . + cp -r /project/ai_forum/computer_vision2/*.py . + ``` + +1. **Setup the kernel for JupyterLab.** You will create a kernel called *grwg_2024_env* to access from JupyterLab Server. Run the following commands to activate the workshop's virtual environment and create a new kernelspec from it: + + {:.copy-code} + ```bash + source /project/ai_forum/computer_vision2/computer_vision_2_env/bin/activate + ipython kernel install --name "computer_vision_2_env" --user + ``` + +1. **Launch a JupyterLab Server session.** Under the *Interactive Apps* menu, select *JupyterLab Server*. Specify the following input values on the page: + + * Account: scinet_workshop1 + * Partition: gpu-a100-mig7 + * QOS: normal 14-00:00:00 + * Number of hours: 4 + * Number of nodes: 1 + * Number of tasks: 2 + * Additional Slurm Parameters: \-\-reservation=forum-gpu --gres=gpu:1 --mem=32G + * Working Directory: /90daydata/shared/${USER} + + Click *Launch*. The screen will update to the *Interactive Sessions* page. When your Jupyter session is ready, the top card will update from *Queued* to *Running* and a *Connect to JupyterLab Server* button will appear. Click *Connect to JupyterLab Server*. + +1. **Select the `computer_vision_2_env` kernel** for the `computer_vision_2.ipynb` notebook. + + diff --git a/sn_collections/_workshops/2024-ai-user-forum/21-computer-vision.md b/sn_collections/_workshops/2024-ai-user-forum/21-computer-vision.md index ca183d46e..16f44f5b3 100644 --- a/sn_collections/_workshops/2024-ai-user-forum/21-computer-vision.md +++ b/sn_collections/_workshops/2024-ai-user-forum/21-computer-vision.md @@ -14,3 +14,45 @@ prerequisites: --- This workshop will teach participants the concepts and tools they need to begin applying modern, deep learning-based computer vision methods to their own scientific research. This will be an interactive, hands-on workshop that will offer plenty of opportunities for practice and experiential learning. By the end of the session, participants will have trained and evaluated a state-of-the-art image classification model on a custom image dataset. + +## Tutorial setup instruction + +Steps to prepare for the tutorial: + +1. **Login to Atlas Open OnDemand** at [https://atlas-ood.hpc.msstate.edu/](https://atlas-ood.hpc.msstate.edu/). For more information on login procedures for web-based SCINet access, see the [SCINet access user guide]({{site.baseurl}}/guides/access/web-based-login). + +1. **Open a command-line session** by clicking on “Clusters” -> “Atlas Shell Access” on the top menu. This will open a new tab with a command-line session on Atlas's login node. + +1. **Create and/or update your workshop working directory** and copy the tutorial materials into it by running the following commands. Note: you do not have to edit the commands with your username as it will be determined by the `$USER` variable. + + {:.copy-code} + ```bash + mkdir -p /90daydata/shared/$USER/computer_vision1 + cd /90daydata/shared/$USER/computer_vision1 + cp -r /project/ai_forum/computer_vision1/computer_vision_1.ipynb . + cp -r /project/ai_forum/computer_vision1/*.py . + ``` + +1. **Setup the kernel for JupyterLab.** You will create a kernel called *grwg_2024_env* to access from JupyterLab Server. Run the following commands to activate the workshop's virtual environment and create a new kernelspec from it: + + {:.copy-code} + ```bash + source /project/ai_forum/computer_vision1/computer_vision_1_env/bin/activate + ipython kernel install --name "computer_vision_1_env" --user + ``` + +1. **Launch a JupyterLab Server session.** Under the *Interactive Apps* menu, select *JupyterLab Server*. Specify the following input values on the page: + + * Account: scinet_workshop1 + * Partition: gpu-a100-mig7 + * QOS: normal 14-00:00:00 + * Number of hours: 4 + * Number of nodes: 1 + * Number of tasks: 2 + * Additional Slurm Parameters: \-\-reservation=forum-gpu --gres=gpu:1 --mem=32G + * Working Directory: /90daydata/shared/${USER} + + Click *Launch*. The screen will update to the *Interactive Sessions* page. When your Jupyter session is ready, the top card will update from *Queued* to *Running* and a *Connect to JupyterLab Server* button will appear. Click *Connect to JupyterLab Server*. + +1. **Select the `computer_vision_1_env` kernel** for the `computer_vision_1.ipynb` notebook. + From c9f23fa8f5a9ded70e8c16bdd76fde548d96161c Mon Sep 17 00:00:00 2001 From: Heather Savoy <47045484+HeatherSavoy-USDA@users.noreply.github.com> Date: Tue, 12 Nov 2024 22:43:16 -0700 Subject: [PATCH 2/5] add --ntasks-per-node and fix env name --- .../_workshops/2024-ai-user-forum/21-computer-vision-2.md | 6 +++--- .../_workshops/2024-ai-user-forum/21-computer-vision.md | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/sn_collections/_workshops/2024-ai-user-forum/21-computer-vision-2.md b/sn_collections/_workshops/2024-ai-user-forum/21-computer-vision-2.md index 6ad1f0da0..0e0130354 100644 --- a/sn_collections/_workshops/2024-ai-user-forum/21-computer-vision-2.md +++ b/sn_collections/_workshops/2024-ai-user-forum/21-computer-vision-2.md @@ -24,7 +24,7 @@ Steps to prepare for the tutorial: 1. **Login to Atlas Open OnDemand** at [https://atlas-ood.hpc.msstate.edu/](https://atlas-ood.hpc.msstate.edu/). For more information on login procedures for web-based SCINet access, see the [SCINet access user guide]({{site.baseurl}}/guides/access/web-based-login). -1. **Open a command-line session** by clicking on “Clusters” -> “Atlas Shell Access” on the top menu. This will open a new tab with a command-line session on Atlas's login node. +1. **Open a command-line session** by clicking on "Clusters" -> "Atlas Shell Access" on the top menu. This will open a new tab with a command-line session on Atlas's login node. 1. **Create and/or update your workshop working directory** and copy the tutorial materials into it by running the following commands. Note: you do not have to edit the commands with your username as it will be determined by the `$USER` variable. @@ -36,7 +36,7 @@ Steps to prepare for the tutorial: cp -r /project/ai_forum/computer_vision2/*.py . ``` -1. **Setup the kernel for JupyterLab.** You will create a kernel called *grwg_2024_env* to access from JupyterLab Server. Run the following commands to activate the workshop's virtual environment and create a new kernelspec from it: +1. **Setup the kernel for JupyterLab.** You will create a kernel called *computer_vision_2_env* to access from JupyterLab Server. Run the following commands to activate the workshop's virtual environment and create a new kernelspec from it: {:.copy-code} ```bash @@ -52,7 +52,7 @@ Steps to prepare for the tutorial: * Number of hours: 4 * Number of nodes: 1 * Number of tasks: 2 - * Additional Slurm Parameters: \-\-reservation=forum-gpu --gres=gpu:1 --mem=32G + * Additional Slurm Parameters: \-\-reservation=forum-gpu --gres=gpu:1 --mem=32G --ntasks-per-node=2 * Working Directory: /90daydata/shared/${USER} Click *Launch*. The screen will update to the *Interactive Sessions* page. When your Jupyter session is ready, the top card will update from *Queued* to *Running* and a *Connect to JupyterLab Server* button will appear. Click *Connect to JupyterLab Server*. diff --git a/sn_collections/_workshops/2024-ai-user-forum/21-computer-vision.md b/sn_collections/_workshops/2024-ai-user-forum/21-computer-vision.md index 16f44f5b3..94978a86b 100644 --- a/sn_collections/_workshops/2024-ai-user-forum/21-computer-vision.md +++ b/sn_collections/_workshops/2024-ai-user-forum/21-computer-vision.md @@ -21,7 +21,7 @@ Steps to prepare for the tutorial: 1. **Login to Atlas Open OnDemand** at [https://atlas-ood.hpc.msstate.edu/](https://atlas-ood.hpc.msstate.edu/). For more information on login procedures for web-based SCINet access, see the [SCINet access user guide]({{site.baseurl}}/guides/access/web-based-login). -1. **Open a command-line session** by clicking on “Clusters” -> “Atlas Shell Access” on the top menu. This will open a new tab with a command-line session on Atlas's login node. +1. **Open a command-line session** by clicking on "Clusters" -> "Atlas Shell Access" on the top menu. This will open a new tab with a command-line session on Atlas's login node. 1. **Create and/or update your workshop working directory** and copy the tutorial materials into it by running the following commands. Note: you do not have to edit the commands with your username as it will be determined by the `$USER` variable. @@ -33,7 +33,7 @@ Steps to prepare for the tutorial: cp -r /project/ai_forum/computer_vision1/*.py . ``` -1. **Setup the kernel for JupyterLab.** You will create a kernel called *grwg_2024_env* to access from JupyterLab Server. Run the following commands to activate the workshop's virtual environment and create a new kernelspec from it: +1. **Setup the kernel for JupyterLab.** You will create a kernel called *computer_vision_1_env* to access from JupyterLab Server. Run the following commands to activate the workshop's virtual environment and create a new kernelspec from it: {:.copy-code} ```bash @@ -49,7 +49,7 @@ Steps to prepare for the tutorial: * Number of hours: 4 * Number of nodes: 1 * Number of tasks: 2 - * Additional Slurm Parameters: \-\-reservation=forum-gpu --gres=gpu:1 --mem=32G + * Additional Slurm Parameters: \-\-reservation=forum-gpu --gres=gpu:1 --mem=32G --ntasks-per-node=2 * Working Directory: /90daydata/shared/${USER} Click *Launch*. The screen will update to the *Interactive Sessions* page. When your Jupyter session is ready, the top card will update from *Queued* to *Running* and a *Connect to JupyterLab Server* button will appear. Click *Connect to JupyterLab Server*. From 75a82e400207e2ac2ef32e9575d37ec1b6931d46 Mon Sep 17 00:00:00 2001 From: Heather Savoy <47045484+HeatherSavoy-USDA@users.noreply.github.com> Date: Wed, 13 Nov 2024 22:48:44 -0700 Subject: [PATCH 3/5] AIUF CV: add interactive job instructions --- .../2024-ai-user-forum/21-computer-vision-2.md | 14 ++++++++++++++ .../2024-ai-user-forum/21-computer-vision.md | 14 ++++++++++++++ 2 files changed, 28 insertions(+) diff --git a/sn_collections/_workshops/2024-ai-user-forum/21-computer-vision-2.md b/sn_collections/_workshops/2024-ai-user-forum/21-computer-vision-2.md index 0e0130354..453815964 100644 --- a/sn_collections/_workshops/2024-ai-user-forum/21-computer-vision-2.md +++ b/sn_collections/_workshops/2024-ai-user-forum/21-computer-vision-2.md @@ -26,6 +26,13 @@ Steps to prepare for the tutorial: 1. **Open a command-line session** by clicking on "Clusters" -> "Atlas Shell Access" on the top menu. This will open a new tab with a command-line session on Atlas's login node. +1. **Request resources on a compute node** by running the following command: + + {:.copy-code} + ```bash + srun --reservation=forum -A scinet_workshop1 -t 00:30:00 -n 1 --mem 8G --pty bash + ``` + 1. **Create and/or update your workshop working directory** and copy the tutorial materials into it by running the following commands. Note: you do not have to edit the commands with your username as it will be determined by the `$USER` variable. {:.copy-code} @@ -44,6 +51,13 @@ Steps to prepare for the tutorial: ipython kernel install --name "computer_vision_2_env" --user ``` +1. **Stop the interactive job** on the compute node by running the command: + + {:.copy-code} + ```bash + exit + ``` + 1. **Launch a JupyterLab Server session.** Under the *Interactive Apps* menu, select *JupyterLab Server*. Specify the following input values on the page: * Account: scinet_workshop1 diff --git a/sn_collections/_workshops/2024-ai-user-forum/21-computer-vision.md b/sn_collections/_workshops/2024-ai-user-forum/21-computer-vision.md index 94978a86b..21a36cf1c 100644 --- a/sn_collections/_workshops/2024-ai-user-forum/21-computer-vision.md +++ b/sn_collections/_workshops/2024-ai-user-forum/21-computer-vision.md @@ -23,6 +23,13 @@ Steps to prepare for the tutorial: 1. **Open a command-line session** by clicking on "Clusters" -> "Atlas Shell Access" on the top menu. This will open a new tab with a command-line session on Atlas's login node. +1. **Request resources on a compute node** by running the following command: + + {:.copy-code} + ```bash + srun --reservation=forum -A scinet_workshop1 -t 00:30:00 -n 1 --mem 8G --pty bash + ``` + 1. **Create and/or update your workshop working directory** and copy the tutorial materials into it by running the following commands. Note: you do not have to edit the commands with your username as it will be determined by the `$USER` variable. {:.copy-code} @@ -41,6 +48,13 @@ Steps to prepare for the tutorial: ipython kernel install --name "computer_vision_1_env" --user ``` +1. **Stop the interactive job** on the compute node by running the command: + + {:.copy-code} + ```bash + exit + ``` + 1. **Launch a JupyterLab Server session.** Under the *Interactive Apps* menu, select *JupyterLab Server*. Specify the following input values on the page: * Account: scinet_workshop1 From 4c420f6c7b1e95a0c32834ad0b865a669c98041a Mon Sep 17 00:00:00 2001 From: Heather Savoy <47045484+HeatherSavoy-USDA@users.noreply.github.com> Date: Thu, 14 Nov 2024 11:39:04 -0700 Subject: [PATCH 4/5] AIUF CV: fix mem parameter --- .../_workshops/2024-ai-user-forum/21-computer-vision-2.md | 2 +- .../_workshops/2024-ai-user-forum/21-computer-vision.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/sn_collections/_workshops/2024-ai-user-forum/21-computer-vision-2.md b/sn_collections/_workshops/2024-ai-user-forum/21-computer-vision-2.md index 453815964..d4252950c 100644 --- a/sn_collections/_workshops/2024-ai-user-forum/21-computer-vision-2.md +++ b/sn_collections/_workshops/2024-ai-user-forum/21-computer-vision-2.md @@ -66,7 +66,7 @@ Steps to prepare for the tutorial: * Number of hours: 4 * Number of nodes: 1 * Number of tasks: 2 - * Additional Slurm Parameters: \-\-reservation=forum-gpu --gres=gpu:1 --mem=32G --ntasks-per-node=2 + * Additional Slurm Parameters: \-\-reservation=forum-gpu \-\-gres=gpu:1 \-\-mem=32G \-\-ntasks-per-node=2 * Working Directory: /90daydata/shared/${USER} Click *Launch*. The screen will update to the *Interactive Sessions* page. When your Jupyter session is ready, the top card will update from *Queued* to *Running* and a *Connect to JupyterLab Server* button will appear. Click *Connect to JupyterLab Server*. diff --git a/sn_collections/_workshops/2024-ai-user-forum/21-computer-vision.md b/sn_collections/_workshops/2024-ai-user-forum/21-computer-vision.md index 21a36cf1c..6ac218b63 100644 --- a/sn_collections/_workshops/2024-ai-user-forum/21-computer-vision.md +++ b/sn_collections/_workshops/2024-ai-user-forum/21-computer-vision.md @@ -63,7 +63,7 @@ Steps to prepare for the tutorial: * Number of hours: 4 * Number of nodes: 1 * Number of tasks: 2 - * Additional Slurm Parameters: \-\-reservation=forum-gpu --gres=gpu:1 --mem=32G --ntasks-per-node=2 + * Additional Slurm Parameters: \-\-reservation=forum-gpu \-\-gres=gpu:1 \-\-mem=32G \-\-ntasks-per-node=2 * Working Directory: /90daydata/shared/${USER} Click *Launch*. The screen will update to the *Interactive Sessions* page. When your Jupyter session is ready, the top card will update from *Queued* to *Running* and a *Connect to JupyterLab Server* button will appear. Click *Connect to JupyterLab Server*. From 09354a00780fa72882885c68bedbf6772cff8053 Mon Sep 17 00:00:00 2001 From: Heather Savoy <47045484+HeatherSavoy-USDA@users.noreply.github.com> Date: Thu, 21 Nov 2024 01:25:48 -0600 Subject: [PATCH 5/5] update CV workshop instructions --- .../21-computer-vision-2.md | 20 ++++++++++++++----- .../2024-ai-user-forum/21-computer-vision.md | 19 ++++++++++++++---- 2 files changed, 30 insertions(+), 9 deletions(-) diff --git a/sn_collections/_workshops/2024-ai-user-forum/21-computer-vision-2.md b/sn_collections/_workshops/2024-ai-user-forum/21-computer-vision-2.md index d4252950c..99d97014d 100644 --- a/sn_collections/_workshops/2024-ai-user-forum/21-computer-vision-2.md +++ b/sn_collections/_workshops/2024-ai-user-forum/21-computer-vision-2.md @@ -66,11 +66,21 @@ Steps to prepare for the tutorial: * Number of hours: 4 * Number of nodes: 1 * Number of tasks: 2 - * Additional Slurm Parameters: \-\-reservation=forum-gpu \-\-gres=gpu:1 \-\-mem=32G \-\-ntasks-per-node=2 - * Working Directory: /90daydata/shared/${USER} + * Additional Slurm Parameters: + + {: .copy-code } + ``` +--reservation=forum-gpu --gres=gpu:1 --mem=32G --ntasks-per-node=2 +``` + * Working Directory: + + {: .copy-code } + ``` +/90daydata/shared/${USER}/computer_vision2 +``` Click *Launch*. The screen will update to the *Interactive Sessions* page. When your Jupyter session is ready, the top card will update from *Queued* to *Running* and a *Connect to JupyterLab Server* button will appear. Click *Connect to JupyterLab Server*. -1. **Select the `computer_vision_2_env` kernel** for the `computer_vision_2.ipynb` notebook. - - +1. **Open the `computer_vision_2.ipynb` notebook.** + +1. **Select the `computer_vision_2_env` kernel** for the notebook. diff --git a/sn_collections/_workshops/2024-ai-user-forum/21-computer-vision.md b/sn_collections/_workshops/2024-ai-user-forum/21-computer-vision.md index 6ac218b63..0d4cc48f5 100644 --- a/sn_collections/_workshops/2024-ai-user-forum/21-computer-vision.md +++ b/sn_collections/_workshops/2024-ai-user-forum/21-computer-vision.md @@ -63,10 +63,21 @@ Steps to prepare for the tutorial: * Number of hours: 4 * Number of nodes: 1 * Number of tasks: 2 - * Additional Slurm Parameters: \-\-reservation=forum-gpu \-\-gres=gpu:1 \-\-mem=32G \-\-ntasks-per-node=2 - * Working Directory: /90daydata/shared/${USER} + * Additional Slurm Parameters: + + {: .copy-code } + ``` +--reservation=forum-gpu --gres=gpu:1 --mem=32G --ntasks-per-node=2 +``` + * Working Directory: + + {: .copy-code } + ``` +/90daydata/shared/${USER}/computer_vision1 +``` Click *Launch*. The screen will update to the *Interactive Sessions* page. When your Jupyter session is ready, the top card will update from *Queued* to *Running* and a *Connect to JupyterLab Server* button will appear. Click *Connect to JupyterLab Server*. -1. **Select the `computer_vision_1_env` kernel** for the `computer_vision_1.ipynb` notebook. - +1. **Open the `computer_vision_1.ipynb` notebook.** + +1. **Select the `computer_vision_1_env` kernel** for the notebook.