Release 4.1.1 merge (#450)

* image correction * access method clarifications * gis tutorial link * reprojected * code and wording updates * plot updates * s3 adjustment * minor feedback updates * added queue info draft 1 * updated table with AWS instances * Added hide and show che side panel FAQ entry * Added intro paragraphs to each major section of the docs * dataset * Some clarifications - thank you for the feedback wildintellect * fix: update conf.py #417 This should resolve the RTD update compatibility. * Make image grid smaller * Reduced size of more content * v4.1.1 (#446) Co-authored-by: bsatoriu <[email protected]> * typo fix (#448) Co-authored-by: bsatoriu <[email protected]> * typo fix (#449) Co-authored-by: bsatoriu <[email protected]> --------- Co-authored-by: smk0033 <[email protected]> Co-authored-by: Rob Tapella <[email protected]> Co-authored-by: Rob Tapella <[email protected]> Co-authored-by: Sheyenne Kirkland <[email protected]> Co-authored-by: Alex I. Mandel <[email protected]> Co-authored-by: bsatoriu <[email protected]>
MAAP-Project · Oct 23, 2024 · 1327e81 · 1327e81
1 parent b6edbc1
commit 1327e81
Show file tree

Hide file tree

Showing 19 changed files with 546 additions and 4,774 deletions.
diff --git a/docs/source/conf.py b/docs/source/conf.py
@@ -10,7 +10,7 @@
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use os.path.abspath to make it absolute, like shown here.
 #
-# import os
+import os
 # import sys
 # sys.path.insert(0, os.path.abspath('.'))
 
@@ -71,3 +71,12 @@ def setup(app):
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
 html_static_path = ['_static']
+
+# Read The Docs Configuration
+# https://about.readthedocs.com/blog/2024/07/addons-by-default/
+html_baseurl = os.environ.get("READTHEDOCS_CANONICAL_URL", "")
+
+if os.environ.get("READTHEDOCS", "") == "True":
+	if "html_context" not in globals():
+        	html_context = {}
+	html_context["READTHEDOCS"] = True
diff --git a/docs/source/getting_started.rst b/docs/source/getting_started.rst
@@ -1,10 +1,12 @@
 Getting Started
 =======================================
 
+This Guide is intended as an initial walk-through of the MAAP platform from registering an account to running an Algorithm in the Data Processing System (DPS). It gives a brief overview of many of the capabilities and concepts of the MAAP platform.
+
 .. toctree::
   :maxdepth: 2
   :caption: Getting Started:
-
+    
   getting_started/about_maap.ipynb
   getting_started/maap_overview.ipynb
   getting_started/getting_started.ipynb

diff --git a/docs/source/getting_started/running_at_scale.ipynb b/docs/source/getting_started/running_at_scale.ipynb
@@ -144,7 +144,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Conclusion\n",
+    "# Conclusion\n",
     "\n",
     "At this point you have gone through the basic steps of setting up and using the MAAP to register and execute an Algorithm in the DPS. This is an example of the first iteration of an algorithm development process that includes writing code, registering it, testing it, making modifications (re-writing code) and re-registering it, and so on.\n",
     "\n",

diff --git a/docs/source/release_notes.rst b/docs/source/release_notes.rst
@@ -1,6 +1,20 @@
 Release Notes
 =======================================
 
+Release notes will mention the release date, a summary for each release, and then comprehensive descrptions of major changes, minor changes, feature removal, and bug-fixes. This is typically more comprehensive than the announcements on the GitHub Discussion board: https://github.com/orgs/MAAP-Project/discussions/categories/announcements.
+
+-------------------------------------------------------------
+4.1.1
+-------------------------------------------------------------
+| October 23, 2024
+| Minor release with non-breaking changes. To use the new features please start a new v4.1.1 workspace.
+
+Fixed
+^^^^^^^^^^^^^^^^^^^
+* Removed pygeos from all base images.
+* Increased number of records in the Jupyter Jobs UI to 200. 
+* Expanded s3 permissions for the maap-py `aws.workspace_bucket_credentials` function. Credentials for user bucket folders now include full read/write access. See `related documentation <system_reference_guide/accessing_bucket_data.html>`_. 
+
 -------------------------------------------------------------
 4.1.0
 -------------------------------------------------------------

diff --git a/docs/source/science/HLS/HLSL30.ipynb b/docs/source/science/HLS/HLSL30.ipynb
diff --git a/docs/source/science/HLS/hls_image_grid.png b/docs/source/science/HLS/hls_image_grid.png
diff --git a/docs/source/science/HLS/hls_map_nvdi_composite.png b/docs/source/science/HLS/hls_map_nvdi_composite.png
diff --git a/docs/source/science/HLS/hls_polygon_maap.png b/docs/source/science/HLS/hls_polygon_maap.png
diff --git a/docs/source/science/NISAR/Simulated_NISAR.ipynb b/docs/source/science/NISAR/Simulated_NISAR.ipynb
diff --git a/docs/source/science_examples.rst b/docs/source/science_examples.rst
@@ -1,6 +1,10 @@
 Science Examples
 =======================================
 
+Here you will find a variety of real-world examples that may help you create a custom Jupyter notebook for your research. These notebooks have dataset specific analysis and visualization examples, and will also provide examples of analysis, data access, tabulation and map-visualization.
+
+You can also find links to Open Source Science guidelines for the MAAP platform.
+
 .. toctree::
   :maxdepth: 2
   :caption: Science Examples:

diff --git a/docs/source/system_reference.rst b/docs/source/system_reference.rst
@@ -1,19 +1,25 @@
 System Reference Guide
 =======================================
 
+This section of documentation is intended as a technical reference for the functionality of the MAAP platform. If you are interested in learning about a particular part of the MAAP platform, this section will contain the canonical reference. 
+
+It is often referenced from the application-focused Science and Technical tutorials.
+
 .. toctree::
   :maxdepth: 2
   :caption: System Reference Guide:
 
   system_reference_guide/create_workspace.ipynb
   system_reference_guide/custom-environments.ipynb
   system_reference_guide/share_data.ipynb
-  system_reference_guide/organizations.ipynb
-  system_reference_guide/ssh.ipynb
   system_reference_guide/algorithm_registration.ipynb
   system_reference_guide/jobsui.ipynb
   system_reference_guide/jobs_maappy.ipynb
+  system_reference_guide/dps_queues.ipynb
   system_reference_guide/work_with_git.ipynb
+  system_reference_guide/organizations.ipynb
+  system_reference_guide/ssh.ipynb
+  system_reference_guide/accessing_bucket_data.ipynb
   system_reference_guide/ade_custom_extensions.rst
   system_reference_guide/faq.rst
 
diff --git a/docs/source/system_reference_guide/accessing_bucket_data.ipynb b/docs/source/system_reference_guide/accessing_bucket_data.ipynb
@@ -0,0 +1,53 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Access workspace bucket data with temporary AWS credentials\n",
+    "\n",
+    "When logged into the ADE, temporary s3 credentials can be issued using the maap-py function `maap.aws.workspace_bucket_credentials()`\n",
+    "\n",
+    "This command issues a set of AWS credentials that grant full read/write access to your own user folder within the `maap-ops-workspace` bucket. Access is given to the directory corresponding to your `my-private-bucket` folder within the ADE. \n",
+    "\n",
+    "```python\n",
+    "\n",
+    "import json\n",
+    "from maap.maap import MAAP\n",
+    "maap = MAAP()\n",
+    "\n",
+    "print(json.dumps(maap.aws.workspace_bucket_credentials(), indent=2))\n",
+    ">>> {\n",
+    "  \"aws_access_key_id\": \"...\",\n",
+    "  \"aws_bucket_name\": \"maap-ops-workspace\",\n",
+    "  \"aws_bucket_prefix\": \"maap_user\",\n",
+    "  \"aws_secret_access_key\": \"...\",\n",
+    "  \"aws_session_expiration\": \"...\",\n",
+    "  \"aws_session_token\": \"...\"\n",
+    "}\n",
+    "```"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.7.8"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git a/docs/source/system_reference_guide/dps_queues.ipynb b/docs/source/system_reference_guide/dps_queues.ipynb
@@ -0,0 +1,81 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "58641795-4f3e-40af-9374-b882f2be6eb1",
+   "metadata": {},
+   "source": [
+    "# Resource Queues"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cd3c4962",
+   "metadata": {},
+   "source": [
+    "Resource Queues on the MAAP Data Processing Service are machines on the cloud that will be processing DPS Jobs. These are [EC2](https://aws.amazon.com/ec2/) machines on AWS and are managed by [auto scaling groups](https://docs.aws.amazon.com/autoscaling/ec2/userguide/auto-scaling-groups.html).\n",
+    "\n",
+    "When a user submits a job, a message is sent to the AWS Auto Scaling Group which provisions the appropriate [machine instance](https://aws.amazon.com/ec2/instance-types/) for processing, which takes about two minutes. The name of the queue provides an indication of what resources the machine will have.\n",
+    "\n",
+    "For example:\n",
+    "\n",
+    "- `maap-dps-worker-8gb` will spin a machine that has at max 8gb of RAM, like t3.large\n",
+    "- `maap-dps-worker-16vcpu-32gb` will spin a machine that has 16 cores and 32gb RAM, like c5.4xlarge.\n",
+    "\n",
+    "\n",
+    "## Queue Access and Organizations\n",
+    "\n",
+    "Only job queues assigned to your [organization(s)](organizations.ipynb) will be allowed for use when registering an Algorithm or submitting Jobs. When submitting Jobs via the Jobs UI, only your org-assigned queues are visible in the queue selection menu. Any publicly-accessible queues, such as the sandbox queue, are accessible to all users of the MAAP platform.\n",
+    "\n",
+    "## Public Queues\n",
+    "\n",
+    "Public queues are queues that are available to all MAAP users irrespective of their association with any organization (i.e guest accounts). Users who belong to an organization and have access to private queues are recommended to use the private queues.\n",
+    "Currently available public queues on the MAAP platform are:\n",
+    "\n",
+    "\n",
+    "| Queue | Memory | Time limit | Guest access | [AWS Instance Type](https://aws.amazon.com/ec2/instance-types/) |\n",
+    "| --- | --- | --- | --- | --- |\n",
+    "| `maap-dps-sandbox` | 8gb | 10 min | yes | t3.large or t3a.large |\n",
+    "| `maap-dps-worker-8gb` | 8gb | unlimited | no | t3.large or t3a.large |\n",
+    "| `maap-dps-worker-16gb` | 16gb | unlimited | no | t3.xlarge or t3a.xlarge |\n",
+    "| `maap-dps-worker-32gb` | 32gb | unlimited | no | r5.xlarge or r5a.xlarge |\n",
+    "\n",
+    "\n",
+    "`maap-dps-sandbox` is a queue introduced in v4.1.0 to allow users to quickly test their algorithm in DPS. This has a max job runtime of 10mins and max RAM of 8gb.\n",
+    "\n",
+    "Note: The platform team will start throttling users ability to submit a batch of jobs on public queues to about 10 jobs per hour.\n",
+    "\n",
+    "\n",
+    "## Private Organization Queues\n",
+    "\n",
+    "Users and user-groups doing science on the MAAP are should reach out to the platform team to enable private organizational queues. Features of private queues are:\n",
+    "\n",
+    "- 24 hr time limit on individual jobs (can be increased upon request).\n",
+    "- No job throttling.\n",
+    "- Custom resources can be requested on the queue, like higher CPU, GPU, etc.\n",
+    "- Dedicated queue, i.e not shared with other users on the MAAP reducing the amount of time spent waiting in the queued state.\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.1"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/docs/source/system_reference_guide/faq.rst b/docs/source/system_reference_guide/faq.rst
@@ -8,4 +8,5 @@ FAQ
   faq/import_granules.ipynb
   faq/jupyter_server.ipynb
   faq/request_data.ipynb
-  faq/delete_algorithm_from_mas.ipynb
+  faq/delete_algorithm_from_mas.ipynb
+  faq/hideshow_sidebar.ipynb
diff --git a/docs/source/system_reference_guide/faq/_static/hide_che_command_palette.png b/docs/source/system_reference_guide/faq/_static/hide_che_command_palette.png
diff --git a/docs/source/system_reference_guide/faq/_static/hide_che_view_menu.png b/docs/source/system_reference_guide/faq/_static/hide_che_view_menu.png
diff --git a/docs/source/system_reference_guide/faq/hideshow_sidebar.ipynb b/docs/source/system_reference_guide/faq/hideshow_sidebar.ipynb
@@ -0,0 +1,42 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "1afe36c9-6206-4366-b208-dfb4aa12cb31",
+   "metadata": {},
+   "source": [
+    "# Can I Hide the Blue Sidebar?\n",
+    "\n",
+    "You can hide and show the blue sidebar (the Eclipse Che sidebar). To do so, open the Jupyter command palette and choose `Hide Che Side Panel` or `Show Che Side Panel`.\n",
+    "\n",
+    "Opening the Jupyter command palette:\n",
+    "![Open Command Palette](_static/hide_che_view_menu.png)\n",
+    "\n",
+    "You will see options at the top to Hide or Show the Che Side Panel. You can also type in some text to filter the list and more easily select the option you would like using only the keyboard.\n",
+    "![Hide Che Option](_static/hide_che_command_palette.png)\n",
+    "\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.1"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/docs/source/system_reference_guide/organizations.ipynb b/docs/source/system_reference_guide/organizations.ipynb
@@ -23,6 +23,8 @@
     "\n",
     "Only job queues assigned to your organization(s) will be allowed for use when registering an algorithm or submitting jobs. When submitting jobs via the Jobs UI, only your org-assigned queues are visible in the queue selection menu. Any publicly-accessible queues, such as the sandbox queue, are accessible to all users of the MAAP platform.\n",
     "\n",
+    "See the [Queues page](dps_queues.ipynb) for more information about the current queues.\n",
+    "\n",
     "### Viewing Your Organizations\n",
     "\n",
     "```\n",

diff --git a/docs/source/technical_tutorials.rst b/docs/source/technical_tutorials.rst
@@ -1,6 +1,8 @@
 Technical Tutorials
 =======================================
 
+This section of the documentation includes hands-on tutorials with example algorithms and data-sets. It is focused on learning how to use the MAAP platform, and not science-oriented.
+
 .. toctree::
   :maxdepth: 2