Merge pull request #56 from cloudnativegeo/fix/fixes1

fix: Markdown link syntax
cloudnativegeo · Sep 18, 2023 · 5c44a71 · 5c44a71
2 parents 1d2ee6e + a18c208
commit 5c44a71
Show file tree

Hide file tree

Showing 5 changed files with 84 additions and 98 deletions.
diff --git a/_quarto.yml b/_quarto.yml
@@ -1,5 +1,6 @@
 project:
   type: website
+  output-dir: _site
   preview:
     port: 4200
     browser: false
@@ -32,9 +33,10 @@ website:
         text: Overview Slides
       - section: Formats
         contents:
-        - section: Cloud Optimized GeoTIFFs
+        - section: Cloud Optimized GeoTIFFs (COG)
           contents:
           - cloud-optimized-geotiffs/intro.qmd
+          - cloud-optimized-geotiffs/cogs-details.qmd
           - cloud-optimized-geotiffs/cogs-examples.ipynb
         - section: Zarr
           contents:

diff --git a/_thankyous.qmd b/_thankyous.qmd
@@ -1,6 +1,4 @@
----
-title: Our supporters
----
+# Our Supporters
 
 ## Thank you to our supporters
 

diff --git a/cloud-optimized-geotiffs/cogs-details.qmd b/cloud-optimized-geotiffs/cogs-details.qmd
@@ -0,0 +1,41 @@
+---
+title: "Make better GeoTIFFs"
+---
+
+# Make better GeoTIFFs 
+
+In the [COG Intro](intro.qmd) you can see what makes a COG different from non-optimized GeoTIFFs. The rest of this page details additional GeoTIFF information that can be relevant to making your files as useful as possible but not a COG requirement.
+
+## Data Type
+
+**Recommendation** The smallest possible data type, that still represents the data appropriately, should be used. It is not generally recommended to shift data from float to integer by multiplying, a space saving technique, as end users then need to undo this step to use the data. Data compression is preferred, see also [Compression](#compression).
+
+GeoTIFF format supports many data types. The key is that all bands must be of the same data type. Unlike some other formats you can not mix and match integers (whole numbers) and floats (decimal numbers) in the same file. If you have this use case consider splitting files by data type and using a catalog like STAC to keep track of them, or look at other formats like [Zarr](../zarr/intro.qmd).
+
+Scenario: If the COG is intended only for visualization, conversion to 3 band byte will improve performance. 
+
+> GDAL supported Data Types [list](https://gdal.org/drivers/raster/gtiff.html#gtiff-geotiff-file-format)
+
+
+## Compression (aka File Size)
+
+The biggest benefit to compression is on the storage side. It’s always recommended to use a lossless compression method. **Deflate** or **LZW** are the most recommended compression algorithms, there are some choices that depend on the data type and distribution, and if the goal is maximum compression or not. Maximum compression does result in some performance loss.
+
+## No Data
+Setting a no data value makes it clear to users and visualization tools what pixels are not actually data. For visualization this allows these pixels to be easily hidden (transparent). Historically many values have been used, 0, -9999, etc… The key is to make sure the GDAL flag for no data is set. It is also suggested that the smallest negative value be used instead of a random value. For byte and unsigned integers/floats this will be 0, if 0 has meaning in your data use a different value (like the max possible value). Having the right nodata flag set is important for overview generation.
+
+## Projection
+
+Read performance can be greatly impacted by the choice of projection and the particular applications used for dynamic tile serving. Using a known CRS defined in the PROJ database (typically EPSG code) is preferred over custom projections. Load times can be 5-20 times greater when using a custom projection. Whenever applying projections make sure to use WKT2 representation. If using a database of known projections, i.e. EPSG codes, this should be fine, there are known issues around manually setting proj-strings.
+
+## What we don’t know (areas of research)
+
+* The optimum size of data at which splitting across files improves performance as a multi-file dataset instead of a single file.
+* When to recommend particular internal tile sizes
+* Compression impacts on http transfer rates.
+* Support for COG creation in all common Geospatial tools varies.
+
+## Additional Resources
+
+* [An Introduction to Cloud Optimized GeoTIFFS (COGs) Part 1: Overview](https://developers.planet.com/docs/planetschool/an-introduction-to-cloud-optimized-geotiffs-cogs-part-1-overview/)
+* [Do you really want people using your data?](https://medium.com/@_VincentS_/do-you-really-want-people-using-your-data-ec94cd94dc3f)
diff --git a/cloud-optimized-geotiffs/cogs-examples.ipynb b/cloud-optimized-geotiffs/cogs-examples.ipynb
@@ -173,6 +173,14 @@
     "veg_gtiff_filename = f\"{test_data_dir}/{veg_files[0]}\""
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "13da6953",
+   "metadata": {},
+   "source": [
+    "> To learn more about the example data see the [Vegetation Continuous Fields (VCF) information page](https://lpdaac.usgs.gov/products/vcf5kyrv001/)."
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "451dbe01",
@@ -297,17 +305,13 @@
    "id": "ff2cc531",
    "metadata": {},
    "source": [
-    "## Dimensions\n",
+    "## Data Structure\n",
     "\n",
-    "This attribute is also sometimes called **chunks** or **internal tiles**.\n",
+    "**Dimensions**\n",
+    "Dimensions are the number of bands, rows and columns stored in a GeoTIFF. [More Info](intro.qmd#dimensions)\n",
     "\n",
-    "Dimensions are the number of bands, rows and columns stored in a GeoTIFF. There is a tradeoff between storing lots of data in one GeoTIFF and storing less data in many GeoTIFFs. The larger a single file, the larger the GeoTIFF header and the multiple requests may be required just to read the spatial index before data retrieval. The opposite problem occurs if you make too many small files, then it takes many reads to retrieve data, and when rendering a combined visualization can greatly impact load time.\n",
-    "\n",
-    "If you plan to pan and zoom a large amount of data through a tiling service in a web browser, there is a tradeoff between 1 large file, or many smaller files. The current recommendation is to meet somewhere in the middle, a moderate amount of medium files.\n",
-    "\n",
-    "### Internal Blocks\n",
-    "\n",
-    "Internal blocks are required if the dimensions of data are over 512x512. However you can control the size of the internal blocks. 256x256 or 512x512 are recommended. When displaying data at full resolution, or doing partial reading of data this size will impact the number of reads required. A size of 256 will take less time to read, and read less data outside the desired bounding box, however for reading large parts of a file, it may take more total read requests. Some clients will aggregate neighboring block reads to reduce the total number of requests. \n",
+    "**Internal Blocks** (aka chunks or internal tiles)\n",
+    "Internal blocks are required if the dimensions of data are over 512x512. [More Info](intro.qmd#internal-blocks)\n",
     "\n",
     "Let's check out the dimensions and blocks of our GeoTIFF and Cloud-Optimized GeoTIFF."
    ]
@@ -423,15 +427,15 @@
     "## Overviews\n",
     "\n",
     "Overviews are downsampled (aggregated) data intended for visualization.\n",
-    "The best resampling algorithm depends on the range, type, and distribution of the data.\n",
     "\n",
-    "The smallest size overview should match the tiling components’ fetch size, typically 256x256. Due to aspect ratio variation just aim to have at least one dimension at or slightly less than 256. The COG driver in GDAL, or rio cogeo tools should do this.\n",
+    "The smallest size overview should match the tiling components’ fetch size, typically 256x256. Due to aspect ratio variation just aim to have at least one dimension at or slightly less than 256. \n",
+    "> The COG driver in GDAL, or rio cogeo tools should do this.\n",
     "\n",
-    "There are many resampling algorithms for generating overviews. When creating overviews several options should be compared before deciding which resampling method to apply.\n",
+    "There are many resampling algorithms for generating overviews. The best resampling algorithm depends on the range, type, and distribution of the data. When creating overviews several options should be compared before deciding which resampling method to apply. \n",
     "\n",
     "GDAL >= 3.2 allows for the overview resampling method to be set directly.\n",
     "\n",
-    "TODO: need to add hints on how to check which resampling method to use for overviews. Possibly provide code for comparing."
+    "<!-- TODO: need to add hints on how to check which resampling method to use for overviews. Possibly provide code for comparing. Alex has a draft of this to add.-->"
    ]
   },
   {
@@ -728,64 +732,11 @@
    "source": [
     "show_overviews(tmp_cog)"
    ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "0d780549-1ffb-45cd-a7ac-969072d46137",
-   "metadata": {},
-   "source": [
-    "# Data Type\n",
-    "\n",
-    "The smallest possible data type, that still represents the data appropriately, should be used. It is not generally recommended to shift data from float to integer by multiplying, a space saving technique, as end users then need to undo this step to use the data. Data compression is preferred, see also Compression.\n",
-    "\n",
-    "Scenario: If the COG is intended only for visualization, conversion to 3 band byte will improve performance. \n",
-    "\n",
-    "# Compression (aka File Size)\n",
-    "\n",
-    "The biggest benefit to compression is on the storage side. It’s always recommended to use a lossless compression method. Deflate or LZW are the most recommended compression algorithms, there are some choices that depend on the data type and distribution, and if the goal is maximum compression or not. Maximum compression does result in some performance loss."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "9fbe71f0-20b9-4a1b-8e7f-de52a90cc7c6",
-   "metadata": {},
-   "source": [
-    "# No Data\n",
-    "Setting a no data value makes it clear to users and visualization tools what pixels are not actually data. For visualization this allows these pixels to be easily hidden (transparent). Historically many values have been used, 0, -9999, etc… The key is to make sure the GDAL flag for no data is set. It is also suggested that the smallest negative value be used instead of a random value. For byte and unsigned integers/floats this will be 0, if 0 has meaning in your data use a different value (like the max possible value). Having the right nodata flag set is important for overview generation.\n",
-    "\n",
-    "# Projection\n",
-    "\n",
-    "Read performance can be greatly impacted by the choice of projection and the particular applications used for dynamic tile serving. Using a known CRS defined in the PROJ database (typically EPSG code) is preferred over custom projections. Load times can be 5-20 times greater when using a custom projection. Whenever applying projections make sure to use WKT2 representation. If using a database of known projections this should be fine, there are known issues around manually setting proj-strings.\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "1e9d49cc-7e1a-4e7b-94a3-539b8e51f0c7",
-   "metadata": {},
-   "source": [
-    "## What we don’t know (areas of research)\n",
-    "\n",
-    "* The optimum size of data at which splitting across files improves performance as a multi-file dataset instead of a single file.\n",
-    "* When to recommend particular internal tile sizes\n",
-    "* Compression impacts on http transfer rates.\n",
-    "* Support for COG creation in other common scientific platforms (e.g. R)\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "d193ab02-bb69-455e-9b72-5b89728f086e",
-   "metadata": {},
-   "source": [
-    "## Additional Resources\n",
-    "\n",
-    "* [An Introduction to Cloud Optimized GeoTIFFS (COGs) Part 1: Overview](https://developers.planet.com/docs/planetschool/an-introduction-to-cloud-optimized-geotiffs-cogs-part-1-overview/)\n",
-    "* [Do you really want people using your data?](https://medium.com/@_VincentS_/do-you-really-want-people-using-your-data-ec94cd94dc3f)"
-   ]
   }
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
+   "display_name": "Python 3.10.12 64-bit",
    "language": "python",
    "name": "python3"
   },
@@ -799,7 +750,12 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.4"
+   "version": "3.10.12"
+  },
+  "vscode": {
+   "interpreter": {
+    "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
+   }
   }
  },
  "nbformat": 4,

diff --git a/cloud-optimized-geotiffs/intro.qmd b/cloud-optimized-geotiffs/intro.qmd
@@ -1,48 +1,37 @@
-# Cloud-Optimized GeoTIFFs
+---
+title: "Cloud-Optimized GeoTIFFs"
+---
 
-## What is a Cloud-Optimized GeoTIFF?
+# What is a Cloud-Optimized GeoTIFF?
 
-Cloud-Optimized GeoTIFF (the COG) is a variant of the TIFF image format that specifies a particular layout of internal data in the GeoTIFF specification to allow for optimized (subsetted or aggregated) access over a network for display or data reading. The key components are overviews, and internal tiling.
+Cloud-Optimized GeoTIFF (the COG), a raster format, is a variant of the TIFF image format that specifies a particular layout of internal data in the GeoTIFF specification to allow for optimized (subsetted or aggregated) access over a network for display or data reading. The key components are overviews, and internal tiling.
 
-For more details see https://www.cogeo.org/
+For more details see [https://www.cogeo.org/](https://www.cogeo.org/)
 
 <img alt="COG Diagram" src="../images/cog-diagram-1.png" width=300/>
 
-### Dimensions
-
-This attribute is also sometimes called **chunks** or **internal tiles**.
+## Dimensions
 
 Dimensions are the number of bands, rows and columns stored in a GeoTIFF. There is a tradeoff between storing lots of data in one GeoTIFF and storing less data in many GeoTIFFs. The larger a single file, the larger the GeoTIFF header and the multiple requests may be required just to read the spatial index before data retrieval. The opposite problem occurs if you make too many small files, then it takes many reads to retrieve data, and when rendering a combined visualization can greatly impact load time.
 
 If you plan to pan and zoom a large amount of data through a tiling service in a web browser, there is a tradeoff between 1 large file, or many smaller files. The current recommendation is to meet somewhere in the middle, a moderate amount of medium files.
 
-### Internal Blocks
+## Internal Blocks
+
+> This attribute is also sometimes called **chunks** or **internal tiles**.
 
 Internal blocks are required if the dimensions of data are over 512x512. However you can control the size of the internal blocks. 256x256 or 512x512 are recommended. When displaying data at full resolution, or doing partial reading of data this size will impact the number of reads required. A size of 256 will take less time to read, and read less data outside the desired bounding box, however for reading large parts of a file, it may take more total read requests. Some clients will aggregate neighboring block reads to reduce the total number of requests.
 
-### Overviews
+## Overviews
 
 Overviews are downsampled (aggregated) data intended for visualization.
 The best resampling algorithm depends on the range, type, and distribution of the data.
 
 The smallest size overview should match the tiling components’ fetch size, typically 256x256. Due to aspect ratio variation just aim to have at least one dimension at or slightly less than 256. The COG driver in GDAL, or rio cogeo tools should do this.
 
-There are many resampling algorithms for generating overviews. When creating overviews several options should be compared before deciding which resampling method to apply.
-
-## How to create and validate COGs
-
-1. [Rio-cogeo: GitHub - cogeotiff/rio-cogeo: Cloud Optimized GeoTIFF creation and validation plugin for rasterio](https://github.com/cogeotiff/rio-cogeo)
-2. [Gdal: COG – Cloud Optimized GeoTIFF generator — GDAL documentation](https://gdal.org/drivers/raster/cog.html)
-
-
-## Additional Resources
-
-* [Planet Blog: An Introduction to Cloud Optimized GeoTIFFS (COGs) Part 1: Overview](https://developers.planet.com/docs/planetschool/an-introduction-to-cloud-optimized-geotiffs-cogs-part-1-overview/)
-* [COG Talk — Part 1: What’s new?](https://medium.com/devseed/cog-talk-part-1-whats-new-941facbcd3d1)
-* [Development Seed Blog: Do you really want people using your data?](https://medium.com/@_VincentS_/do-you-really-want-people-using-your-data-ec94cd94dc3f)
+There are many [resampling algorithms](https://gdal.org/programs/gdal_translate.html#cmdoption-gdal_translate-r) for generating overviews. When creating overviews several options should be compared before deciding which resampling method to apply.
 
-## How to visualize COGs
+## See more
 
-* GDAL vis* drivers (vsicurl, vsis3, vsiaz,)
-* Titiler https://github.com/developmentseed/titiler
-* Rio-viz https://github.com/developmentseed/rio-viz
+- Additional [GeoTIFF details](cogs-details.qmd) that can be helpful.
+- Making and using [COG examples](cogs-examples.ipynb).