From d7036b555383bad81ca17c6bbd98f0352cb05021 Mon Sep 17 00:00:00 2001 From: SammyAgrawal Date: Thu, 27 Jun 2024 14:58:27 -0400 Subject: [PATCH 1/7] added section on user directory --- book/guides/faq.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/book/guides/faq.md b/book/guides/faq.md index d1fc26b..355b783 100644 --- a/book/guides/faq.md +++ b/book/guides/faq.md @@ -21,3 +21,11 @@ If you **are** member of one of the github teams, ask them to try the following - \[ \] Try a different browser - \[ \] Restart the computer - If these steps do not work, please reach out to the [](support.data_compute_team). + +## I received a warning about space on my User Directory +To accommodate an expanding LEAP community, the data and compute team has instituted a storage quota for all [User Directories](https://leap-stc.github.io/leap-pangeo/jupyterhub.html#your-user-directory). We hope a recommended limit of 25GB and hard limit of 50GB will help maintain the reliability of the hub for all users (recall that everyone is using the same underlying storage so if it fills up, the Hub crashes for everyone!). If you get a Hub Usage Alert email, this means you are going beyond the recommended limits. Users who persistently violate hub usage policies may temporarily get reduced access. Some ways to address this are: +- consider moving completed or old work to alternate storage locations such as Github. +- Deleting cached files and ipython checkpoints from old projects +- move large datasets into a [cloud bucket](https://leap-stc.github.io/leap-pangeo/jupyterhub.html#leap-pangeo-cloud-storage-buckets). Remember that user directories are for scripts and notebooks not datasets! + +Our goal is to accomodate all users and thus we are happy to assist users in relocating data. If you have any concerns, please reach out to the [](support.data_compute_team). \ No newline at end of file From 4c8601b5af902b5be1cd29d3e66558360c33b872 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Thu, 27 Jun 2024 19:01:04 +0000 Subject: [PATCH 2/7] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- book/guides/faq.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/book/guides/faq.md b/book/guides/faq.md index 355b783..fbc850b 100644 --- a/book/guides/faq.md +++ b/book/guides/faq.md @@ -23,9 +23,11 @@ If you **are** member of one of the github teams, ask them to try the following - If these steps do not work, please reach out to the [](support.data_compute_team). ## I received a warning about space on my User Directory -To accommodate an expanding LEAP community, the data and compute team has instituted a storage quota for all [User Directories](https://leap-stc.github.io/leap-pangeo/jupyterhub.html#your-user-directory). We hope a recommended limit of 25GB and hard limit of 50GB will help maintain the reliability of the hub for all users (recall that everyone is using the same underlying storage so if it fills up, the Hub crashes for everyone!). If you get a Hub Usage Alert email, this means you are going beyond the recommended limits. Users who persistently violate hub usage policies may temporarily get reduced access. Some ways to address this are: -- consider moving completed or old work to alternate storage locations such as Github. + +To accommodate an expanding LEAP community, the data and compute team has instituted a storage quota for all [User Directories](https://leap-stc.github.io/leap-pangeo/jupyterhub.html#your-user-directory). We hope a recommended limit of 25GB and hard limit of 50GB will help maintain the reliability of the hub for all users (recall that everyone is using the same underlying storage so if it fills up, the Hub crashes for everyone!). If you get a Hub Usage Alert email, this means you are going beyond the recommended limits. Users who persistently violate hub usage policies may temporarily get reduced access. Some ways to address this are: + +- consider moving completed or old work to alternate storage locations such as Github. - Deleting cached files and ipython checkpoints from old projects -- move large datasets into a [cloud bucket](https://leap-stc.github.io/leap-pangeo/jupyterhub.html#leap-pangeo-cloud-storage-buckets). Remember that user directories are for scripts and notebooks not datasets! +- move large datasets into a [cloud bucket](https://leap-stc.github.io/leap-pangeo/jupyterhub.html#leap-pangeo-cloud-storage-buckets). Remember that user directories are for scripts and notebooks not datasets! -Our goal is to accomodate all users and thus we are happy to assist users in relocating data. If you have any concerns, please reach out to the [](support.data_compute_team). \ No newline at end of file +Our goal is to accomodate all users and thus we are happy to assist users in relocating data. If you have any concerns, please reach out to the [](support.data_compute_team). From 1bad4953f91ed5514642cadd31a942f3bd9d688d Mon Sep 17 00:00:00 2001 From: SammyAgrawal Date: Thu, 27 Jun 2024 17:30:29 -0400 Subject: [PATCH 3/7] updated guide for user dir warning --- book/guides/faq.md | 11 ++++++----- book/leap-pangeo/jupyterhub.md | 4 ++-- 2 files changed, 8 insertions(+), 7 deletions(-) diff --git a/book/guides/faq.md b/book/guides/faq.md index 355b783..213a3d2 100644 --- a/book/guides/faq.md +++ b/book/guides/faq.md @@ -23,9 +23,10 @@ If you **are** member of one of the github teams, ask them to try the following - If these steps do not work, please reach out to the [](support.data_compute_team). ## I received a warning about space on my User Directory -To accommodate an expanding LEAP community, the data and compute team has instituted a storage quota for all [User Directories](https://leap-stc.github.io/leap-pangeo/jupyterhub.html#your-user-directory). We hope a recommended limit of 25GB and hard limit of 50GB will help maintain the reliability of the hub for all users (recall that everyone is using the same underlying storage so if it fills up, the Hub crashes for everyone!). If you get a Hub Usage Alert email, this means you are going beyond the recommended limits. Users who persistently violate hub usage policies may temporarily get reduced access. Some ways to address this are: -- consider moving completed or old work to alternate storage locations such as Github. -- Deleting cached files and ipython checkpoints from old projects -- move large datasets into a [cloud bucket](https://leap-stc.github.io/leap-pangeo/jupyterhub.html#leap-pangeo-cloud-storage-buckets). Remember that user directories are for scripts and notebooks not datasets! +If you get a Hub Usage Alert email, this means you are violating the User Directory storage limit (to learn why this limit exists, see read about [User Directories](hub.guide.data.user_dir)). Users who persistently violate hub usage policies may temporarily get reduced cloud access. +**Troubleshooting** +- To see which files and directories are taking up the bulk of your storage, run `du -h --max-depth=1 ~/ | sort -h` in Terminal. It will likely be cached files and small/medium size data files. +- Delete cached files, ipython checkpoints, and anything else not needed from old projects +- move data into a [cloud bucket](https://leap-stc.github.io/leap-pangeo/jupyterhub.html#leap-pangeo-cloud-storage-buckets). Remember that user directories are for scripts and notebooks not datasets! -Our goal is to accomodate all users and thus we are happy to assist users in relocating data. If you have any concerns, please reach out to the [](support.data_compute_team). \ No newline at end of file +Our goal is to accomodate all users and thus we are happy to assist users in relocating data. If you have any concerns, please reach out to the [](support.data_compute_team). You may also consult our [Data Guide](guide.hub.data) and [Data Policy](./../policies/data_policy). \ No newline at end of file diff --git a/book/leap-pangeo/jupyterhub.md b/book/leap-pangeo/jupyterhub.md index 60d7ebc..19e8a00 100644 --- a/book/leap-pangeo/jupyterhub.md +++ b/book/leap-pangeo/jupyterhub.md @@ -97,11 +97,11 @@ As shown in the picture above, every user will see `'/home/jovyan'` as their roo The primary purpose of this directory is to store small files, like github repositories and other code. :::\{warning} -Please do not store large files in your user directory `/home/jovyan`. Your home directory is intended only for notebooks, analysis scripts, and small datasets (\< 1 GB). It is not an appropriate place to store large datasets. +Please do not store large files in your user directory `/home/jovyan`. Your home directory is intended only for notebooks, analysis scripts, and small datasets (\< 1 GB). It is not an appropriate place to store large datasets. Unlike the cloud buckets, these directories all use the same underlying storage so if a single user fills up the space, the Hub crashes for everyone. To accommodate the expanding LEAP community, the data and compute team has instituted a storage quota. We recommend users use less than 25GB and maintain a hard limit of 50GB. **Users who persistently violate the hard limit may temporarily get reduced cloud access**. To check how much space you are using in your home directory open a terminal window on the hub and run `du -h --max-depth=1 ~/ | sort -h`. -If you want to save larger files for your work use our [](hub.data.buckets) and consult out [Hub Data Guide](guide.hub.data) +If you want to save larger files for your work use our [](hub.data.buckets) and consult our [Hub Data Guide](guide.hub.data) ::: (hub.data.buckets)= From 5e4d7233fc153c77ba29b81f952c31c82432d904 Mon Sep 17 00:00:00 2001 From: SammyAgrawal Date: Thu, 27 Jun 2024 17:48:21 -0400 Subject: [PATCH 4/7] modified faq links and text --- book/guides/faq.md | 9 ++++++--- book/leap-pangeo/jupyterhub.md | 4 ++-- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/book/guides/faq.md b/book/guides/faq.md index 28a28cb..7f365e9 100644 --- a/book/guides/faq.md +++ b/book/guides/faq.md @@ -22,10 +22,13 @@ If you **are** member of one of the github teams, ask them to try the following - \[ \] Restart the computer - If these steps do not work, please reach out to the [](support.data_compute_team). +(faq.usr_dir_usage_warning)= ## I received a warning about space on my User Directory If you get a Hub Usage Alert email, this means you are violating the User Directory storage limit (to learn why this limit exists, see read about [User Directories](hub.guide.data.user_dir)). Users who persistently violate hub usage policies may temporarily get reduced cloud access. + **Troubleshooting** -- To see which files and directories are taking up the bulk of your storage, run `du -h --max-depth=1 ~/ | sort -h` in Terminal. It will likely be cached files and small/medium size data files. +- To see which files and directories are taking up the bulk of your storage, run `du -h --max-depth=1 ~/ | sort -h` in Terminal. It will likely be cached files and small/medium size data files that can be removed without disrupting typical usage. - Delete cached files, ipython checkpoints, and anything else not needed from old projects -- move data into a [cloud bucket](https://leap-stc.github.io/leap-pangeo/jupyterhub.html#leap-pangeo-cloud-storage-buckets). Remember that user directories are for scripts and notebooks not datasets! -Our goal is to accomodate all users and thus we are happy to assist users in relocating data. If you have any concerns, please reach out to the [](support.data_compute_team). You may also consult our [Data Guide](guide.hub.data) and [Data Policy](./../policies/data_policy). +- Move needed data files into a persistent LEAP [cloud bucket](https://leap-stc.github.io/leap-pangeo/jupyterhub.html#leap-pangeo-cloud-storage-buckets). Remember that user directories are for scripts and notebooks not datasets! + +Our goal is to accomodate all community members and thus we are happy to assist users in relocating data. If you have any concerns, please reach out to the [](support.data_compute_team). You may also consult our [Hub Data Guide](guide.hub.data) and [Data Policy](./../policies/data_policy). diff --git a/book/leap-pangeo/jupyterhub.md b/book/leap-pangeo/jupyterhub.md index 19e8a00..ca89014 100644 --- a/book/leap-pangeo/jupyterhub.md +++ b/book/leap-pangeo/jupyterhub.md @@ -97,11 +97,11 @@ As shown in the picture above, every user will see `'/home/jovyan'` as their roo The primary purpose of this directory is to store small files, like github repositories and other code. :::\{warning} -Please do not store large files in your user directory `/home/jovyan`. Your home directory is intended only for notebooks, analysis scripts, and small datasets (\< 1 GB). It is not an appropriate place to store large datasets. Unlike the cloud buckets, these directories all use the same underlying storage so if a single user fills up the space, the Hub crashes for everyone. To accommodate the expanding LEAP community, the data and compute team has instituted a storage quota. We recommend users use less than 25GB and maintain a hard limit of 50GB. **Users who persistently violate the hard limit may temporarily get reduced cloud access**. +To accommodate the expanding LEAP community, the data and compute team has instituted a storage quota on individual user directories `/home/jovyan`. Your home directory is intended only for notebooks, analysis scripts, and small datasets (\< 1 GB). It is not an appropriate place to store large datasets. Unlike the cloud buckets, these directories use an underlying storage with a rigid limit. If a single user fills up the space, the Hub crashes for everyone. We recommend users use less than 25GB and enforce a hard limit of 50GB. **Users who persistently violate the limit may temporarily get reduced cloud access**. To check how much space you are using in your home directory open a terminal window on the hub and run `du -h --max-depth=1 ~/ | sort -h`. -If you want to save larger files for your work use our [](hub.data.buckets) and consult our [Hub Data Guide](guide.hub.data) +If you want to save larger files for your work use our [](hub.data.buckets) and consult our [Hub Data Guide](guide.hub.data). See the [FAQs](faq.usr_dir_usage_warning) for guidance on reducing storage. ::: (hub.data.buckets)= From 0930fe3334af803afa35347acb760556a3704f1d Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Thu, 27 Jun 2024 21:48:56 +0000 Subject: [PATCH 5/7] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- book/guides/faq.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/book/guides/faq.md b/book/guides/faq.md index 7f365e9..8b2cc8b 100644 --- a/book/guides/faq.md +++ b/book/guides/faq.md @@ -23,12 +23,15 @@ If you **are** member of one of the github teams, ask them to try the following - If these steps do not work, please reach out to the [](support.data_compute_team). (faq.usr_dir_usage_warning)= + ## I received a warning about space on my User Directory -If you get a Hub Usage Alert email, this means you are violating the User Directory storage limit (to learn why this limit exists, see read about [User Directories](hub.guide.data.user_dir)). Users who persistently violate hub usage policies may temporarily get reduced cloud access. + +If you get a Hub Usage Alert email, this means you are violating the User Directory storage limit (to learn why this limit exists, see read about [User Directories](hub.guide.data.user_dir)). Users who persistently violate hub usage policies may temporarily get reduced cloud access. **Troubleshooting** -- To see which files and directories are taking up the bulk of your storage, run `du -h --max-depth=1 ~/ | sort -h` in Terminal. It will likely be cached files and small/medium size data files that can be removed without disrupting typical usage. + +- To see which files and directories are taking up the bulk of your storage, run `du -h --max-depth=1 ~/ | sort -h` in Terminal. It will likely be cached files and small/medium size data files that can be removed without disrupting typical usage. - Delete cached files, ipython checkpoints, and anything else not needed from old projects -- Move needed data files into a persistent LEAP [cloud bucket](https://leap-stc.github.io/leap-pangeo/jupyterhub.html#leap-pangeo-cloud-storage-buckets). Remember that user directories are for scripts and notebooks not datasets! +- Move needed data files into a persistent LEAP [cloud bucket](https://leap-stc.github.io/leap-pangeo/jupyterhub.html#leap-pangeo-cloud-storage-buckets). Remember that user directories are for scripts and notebooks not datasets! -Our goal is to accomodate all community members and thus we are happy to assist users in relocating data. If you have any concerns, please reach out to the [](support.data_compute_team). You may also consult our [Hub Data Guide](guide.hub.data) and [Data Policy](./../policies/data_policy). +Our goal is to accomodate all community members and thus we are happy to assist users in relocating data. If you have any concerns, please reach out to the [](support.data_compute_team). You may also consult our [Hub Data Guide](guide.hub.data) and [Data Policy](./../policies/data_policy). From b922b6aad7cd06032ca60d1b6b95d97a49b70b39 Mon Sep 17 00:00:00 2001 From: SammyAgrawal Date: Mon, 1 Jul 2024 14:58:25 -0400 Subject: [PATCH 6/7] reformatted faq --- book/guides/faq.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/book/guides/faq.md b/book/guides/faq.md index 7f365e9..95e542a 100644 --- a/book/guides/faq.md +++ b/book/guides/faq.md @@ -24,11 +24,12 @@ If you **are** member of one of the github teams, ask them to try the following (faq.usr_dir_usage_warning)= ## I received a warning about space on my User Directory -If you get a Hub Usage Alert email, this means you are violating the User Directory storage limit (to learn why this limit exists, see read about [User Directories](hub.guide.data.user_dir)). Users who persistently violate hub usage policies may temporarily get reduced cloud access. +If you get a Hub Usage Alert email, this means you are violating the User Directory storage limit (to learn why this limit exists, see read about [User Directories](hub.guide.data.user_dir)). Remember that user directories are for scripts and notebooks not datasets! Users who persistently violate hub usage policies may temporarily get reduced cloud access. **Troubleshooting** -- To see which files and directories are taking up the bulk of your storage, run `du -h --max-depth=1 ~/ | sort -h` in Terminal. It will likely be cached files and small/medium size data files that can be removed without disrupting typical usage. -- Delete cached files, ipython checkpoints, and anything else not needed from old projects -- Move needed data files into a persistent LEAP [cloud bucket](https://leap-stc.github.io/leap-pangeo/jupyterhub.html#leap-pangeo-cloud-storage-buckets). Remember that user directories are for scripts and notebooks not datasets! +- To see which files and directories are taking up the bulk of your storage, run `du -h --max-depth=1 ~/ | sort -h` in Terminal. It will likely reveal cached files and small/medium size data files that can be removed without disrupting typical usage. +- Delete cached files, ipython checkpoints, and any other unwanted files. +- If you still require more storage, it is likely that you are storing downloaded data in your user directory. We recommend storing data in a LEAP [cloud bucket](https://leap-stc.github.io/leap-pangeo/jupyterhub.html#leap-pangeo-cloud-storage-buckets) or data catalog. For more information, please consult our [Hub Data Guide](guide.hub.data) and [Data Policy](./../policies/data_policy). -Our goal is to accomodate all community members and thus we are happy to assist users in relocating data. If you have any concerns, please reach out to the [](support.data_compute_team). You may also consult our [Hub Data Guide](guide.hub.data) and [Data Policy](./../policies/data_policy). + +Our goal is to accomodate all community members and thus we are happy to assist users in relocating data. If you have any concerns, please reach out to the [](support.data_compute_team). \ No newline at end of file From e5b303a7e22fedf82510a258d80548426acb0369 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Mon, 1 Jul 2024 19:00:16 +0000 Subject: [PATCH 7/7] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- book/guides/faq.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/book/guides/faq.md b/book/guides/faq.md index 980791b..799aca6 100644 --- a/book/guides/faq.md +++ b/book/guides/faq.md @@ -25,12 +25,13 @@ If you **are** member of one of the github teams, ask them to try the following (faq.usr_dir_usage_warning)= ## I received a warning about space on my User Directory -If you get a Hub Usage Alert email, this means you are violating the User Directory storage limit (to learn why this limit exists, see read about [User Directories](hub.guide.data.user_dir)). Remember that user directories are for scripts and notebooks not datasets! Users who persistently violate hub usage policies may temporarily get reduced cloud access. + +If you get a Hub Usage Alert email, this means you are violating the User Directory storage limit (to learn why this limit exists, see read about [User Directories](hub.guide.data.user_dir)). Remember that user directories are for scripts and notebooks not datasets! Users who persistently violate hub usage policies may temporarily get reduced cloud access. **Troubleshooting** -- To see which files and directories are taking up the bulk of your storage, run `du -h --max-depth=1 ~/ | sort -h` in Terminal. It will likely reveal cached files and small/medium size data files that can be removed without disrupting typical usage. -- Delete cached files, ipython checkpoints, and any other unwanted files. -- If you still require more storage, it is likely that you are storing downloaded data in your user directory. We recommend storing data in a LEAP [cloud bucket](https://leap-stc.github.io/leap-pangeo/jupyterhub.html#leap-pangeo-cloud-storage-buckets) or data catalog. For more information, please consult our [Hub Data Guide](guide.hub.data) and [Data Policy](./../policies/data_policy). +- To see which files and directories are taking up the bulk of your storage, run `du -h --max-depth=1 ~/ | sort -h` in Terminal. It will likely reveal cached files and small/medium size data files that can be removed without disrupting typical usage. +- Delete cached files, ipython checkpoints, and any other unwanted files. +- If you still require more storage, it is likely that you are storing downloaded data in your user directory. We recommend storing data in a LEAP [cloud bucket](https://leap-stc.github.io/leap-pangeo/jupyterhub.html#leap-pangeo-cloud-storage-buckets) or data catalog. For more information, please consult our [Hub Data Guide](guide.hub.data) and [Data Policy](./../policies/data_policy). Our goal is to accomodate all community members and thus we are happy to assist users in relocating data. If you have any concerns, please reach out to the [](support.data_compute_team).