From 5afed828665828cdcd4f78227ff40a456e0fdacf Mon Sep 17 00:00:00 2001 From: Kyle Barron Date: Mon, 18 Sep 2023 10:28:11 -0600 Subject: [PATCH 1/3] Convert cogs intro notebook to markdown --- _quarto.yml | 2 +- cloud-optimized-geotiffs/intro.ipynb | 91 ---------------------------- cloud-optimized-geotiffs/intro.qmd | 48 +++++++++++++++ 3 files changed, 49 insertions(+), 92 deletions(-) delete mode 100644 cloud-optimized-geotiffs/intro.ipynb create mode 100644 cloud-optimized-geotiffs/intro.qmd diff --git a/_quarto.yml b/_quarto.yml index b2a8d8a..fcec87b 100644 --- a/_quarto.yml +++ b/_quarto.yml @@ -34,7 +34,7 @@ website: contents: - section: Cloud Optimized GeoTIFFs contents: - - cloud-optimized-geotiffs/intro.ipynb + - cloud-optimized-geotiffs/intro.qmd - cloud-optimized-geotiffs/cogs-examples.ipynb - section: Zarr contents: diff --git a/cloud-optimized-geotiffs/intro.ipynb b/cloud-optimized-geotiffs/intro.ipynb deleted file mode 100644 index e31774a..0000000 --- a/cloud-optimized-geotiffs/intro.ipynb +++ /dev/null @@ -1,91 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "id": "e84fbc0b", - "metadata": {}, - "source": [ - "# Cloud-Optimized GeoTIFFs\n", - "\n", - "## What is a Cloud-Optimized GeoTIFF?\n", - "\n", - "Cloud-Optimized GeoTIFF (the COG) is a variant of the TIFF image format that specifies a particular layout of internal data in the GeoTIFF specification to allow for optimized (subsetted or aggregated) access over a network for display or data reading. The key components are overviews, and internal tiling.\n", - "\n", - "For more details see https://www.cogeo.org/\n", - "\n", - "\"COG\n", - "\n", - "### Dimensions\n", - "\n", - "This attribute is also sometimes called **chunks** or **internal tiles**.\n", - "\n", - "Dimensions are the number of bands, rows and columns stored in a GeoTIFF. There is a tradeoff between storing lots of data in one GeoTIFF and storing less data in many GeoTIFFs. The larger a single file, the larger the GeoTIFF header and the multiple requests may be required just to read the spatial index before data retrieval. The opposite problem occurs if you make too many small files, then it takes many reads to retrieve data, and when rendering a combined visualization can greatly impact load time.\n", - "\n", - "If you plan to pan and zoom a large amount of data through a tiling service in a web browser, there is a tradeoff between 1 large file, or many smaller files. The current recommendation is to meet somewhere in the middle, a moderate amount of medium files.\n", - "\n", - "### Internal Blocks\n", - "\n", - "Internal blocks are required if the dimensions of data are over 512x512. However you can control the size of the internal blocks. 256x256 or 512x512 are recommended. When displaying data at full resolution, or doing partial reading of data this size will impact the number of reads required. A size of 256 will take less time to read, and read less data outside the desired bounding box, however for reading large parts of a file, it may take more total read requests. Some clients will aggregate neighboring block reads to reduce the total number of requests.\n", - "\n", - "### Overviews\n", - "\n", - "Overviews are downsampled (aggregated) data intended for visualization.\n", - "The best resampling algorithm depends on the range, type, and distribution of the data.\n", - "\n", - "The smallest size overview should match the tiling components’ fetch size, typically 256x256. Due to aspect ratio variation just aim to have at least one dimension at or slightly less than 256. The COG driver in GDAL, or rio cogeo tools should do this.\n", - "\n", - "There are many resampling algorithms for generating overviews. When creating overviews several options should be compared before deciding which resampling method to apply." - ] - }, - { - "cell_type": "markdown", - "id": "eb04e257-322a-4d42-b6d6-e75f1c587a69", - "metadata": {}, - "source": [ - "## How to create and validate COGs\n", - "\n", - "1. [Rio-cogeo: GitHub - cogeotiff/rio-cogeo: Cloud Optimized GeoTIFF creation and validation plugin for rasterio](https://github.com/cogeotiff/rio-cogeo)\n", - "2. [Gdal: COG – Cloud Optimized GeoTIFF generator — GDAL documentation](https://gdal.org/drivers/raster/cog.html)" - ] - }, - { - "cell_type": "markdown", - "id": "d193ab02-bb69-455e-9b72-5b89728f086e", - "metadata": {}, - "source": [ - "## Additional Resources\n", - "\n", - "* [Planet Blog: An Introduction to Cloud Optimized GeoTIFFS (COGs) Part 1: Overview](https://developers.planet.com/docs/planetschool/an-introduction-to-cloud-optimized-geotiffs-cogs-part-1-overview/)\n", - "* [COG Talk — Part 1: What’s new?](https://medium.com/devseed/cog-talk-part-1-whats-new-941facbcd3d1)\n", - "* [Development Seed Blog: Do you really want people using your data?](https://medium.com/@_VincentS_/do-you-really-want-people-using-your-data-ec94cd94dc3f)\n", - "\n", - "## How to visualize COGs\n", - "\n", - "* GDAL vis* drivers (vsicurl, vsis3, vsiaz,)\n", - "* Titiler https://github.com/developmentseed/titiler\n", - "* Rio-viz https://github.com/developmentseed/rio-viz" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python [conda env:geospatial]", - "language": "python", - "name": "conda-env-geospatial-py" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.10.6" - } - }, - "nbformat": 4, - "nbformat_minor": 5 -} diff --git a/cloud-optimized-geotiffs/intro.qmd b/cloud-optimized-geotiffs/intro.qmd new file mode 100644 index 0000000..4cf9484 --- /dev/null +++ b/cloud-optimized-geotiffs/intro.qmd @@ -0,0 +1,48 @@ +# Cloud-Optimized GeoTIFFs + +## What is a Cloud-Optimized GeoTIFF? + +Cloud-Optimized GeoTIFF (the COG) is a variant of the TIFF image format that specifies a particular layout of internal data in the GeoTIFF specification to allow for optimized (subsetted or aggregated) access over a network for display or data reading. The key components are overviews, and internal tiling. + +For more details see https://www.cogeo.org/ + +COG Diagram + +### Dimensions + +This attribute is also sometimes called **chunks** or **internal tiles**. + +Dimensions are the number of bands, rows and columns stored in a GeoTIFF. There is a tradeoff between storing lots of data in one GeoTIFF and storing less data in many GeoTIFFs. The larger a single file, the larger the GeoTIFF header and the multiple requests may be required just to read the spatial index before data retrieval. The opposite problem occurs if you make too many small files, then it takes many reads to retrieve data, and when rendering a combined visualization can greatly impact load time. + +If you plan to pan and zoom a large amount of data through a tiling service in a web browser, there is a tradeoff between 1 large file, or many smaller files. The current recommendation is to meet somewhere in the middle, a moderate amount of medium files. + +### Internal Blocks + +Internal blocks are required if the dimensions of data are over 512x512. However you can control the size of the internal blocks. 256x256 or 512x512 are recommended. When displaying data at full resolution, or doing partial reading of data this size will impact the number of reads required. A size of 256 will take less time to read, and read less data outside the desired bounding box, however for reading large parts of a file, it may take more total read requests. Some clients will aggregate neighboring block reads to reduce the total number of requests. + +### Overviews + +Overviews are downsampled (aggregated) data intended for visualization. +The best resampling algorithm depends on the range, type, and distribution of the data. + +The smallest size overview should match the tiling components’ fetch size, typically 256x256. Due to aspect ratio variation just aim to have at least one dimension at or slightly less than 256. The COG driver in GDAL, or rio cogeo tools should do this. + +There are many resampling algorithms for generating overviews. When creating overviews several options should be compared before deciding which resampling method to apply. + +## How to create and validate COGs + +1. [Rio-cogeo: GitHub - cogeotiff/rio-cogeo: Cloud Optimized GeoTIFF creation and validation plugin for rasterio](https://github.com/cogeotiff/rio-cogeo) +2. [Gdal: COG – Cloud Optimized GeoTIFF generator — GDAL documentation](https://gdal.org/drivers/raster/cog.html) + + +## Additional Resources + +* [Planet Blog: An Introduction to Cloud Optimized GeoTIFFS (COGs) Part 1: Overview](https://developers.planet.com/docs/planetschool/an-introduction-to-cloud-optimized-geotiffs-cogs-part-1-overview/) +* [COG Talk — Part 1: What’s new?](https://medium.com/devseed/cog-talk-part-1-whats-new-941facbcd3d1) +* [Development Seed Blog: Do you really want people using your data?](https://medium.com/@_VincentS_/do-you-really-want-people-using-your-data-ec94cd94dc3f) + +## How to visualize COGs + +* GDAL vis* drivers (vsicurl, vsis3, vsiaz,) +* Titiler https://github.com/developmentseed/titiler +* Rio-viz https://github.com/developmentseed/rio-viz From df0284269e644aaa22b2aacdad81f7427b50552f Mon Sep 17 00:00:00 2001 From: Kyle Barron Date: Mon, 18 Sep 2023 10:28:56 -0600 Subject: [PATCH 2/3] fix link --- index.qmd | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/index.qmd b/index.qmd index c38abfd..6626972 100644 --- a/index.qmd +++ b/index.qmd @@ -54,7 +54,7 @@ Notes: 1. [Overview of Formats (slideshow)](./overview.qmd) 2. Formats - a. [Cloud-Optimized GeoTIFFs](./cloud-optimized-geotiffs/intro.ipynb) + a. [Cloud-Optimized GeoTIFFs](./cloud-optimized-geotiffs/intro.qmd) b. [Zarr](./zarr/intro.qmd) c. [Kerchunk](./kerchunk/intro.qmd) c. [Cloud-Optimized NetCDF4/HDF5](./cloud-optimized-netcdf4-hdf5/index.qmd) @@ -82,4 +82,4 @@ Most of the data formats covered in this guide have a Jupyter Notebook example t 4. What is the expected access method? 5. How much of your data is typically rendered or selected at once? -{{< include _thankyous.qmd >}} \ No newline at end of file +{{< include _thankyous.qmd >}} From 3cd2c1604de5009c1bee6d0725515a295b7c91c8 Mon Sep 17 00:00:00 2001 From: Kyle Barron Date: Mon, 18 Sep 2023 10:32:58 -0600 Subject: [PATCH 3/3] change to _site --- .github/workflows/preview.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/preview.yml b/.github/workflows/preview.yml index 0181d34..6796c1c 100644 --- a/.github/workflows/preview.yml +++ b/.github/workflows/preview.yml @@ -24,4 +24,4 @@ jobs: - name: Deploy preview uses: rossjrw/pr-preview-action@v1 with: - source-dir: docs + source-dir: _site